It is not Ubuntu that does this. It is the Linux kernel that does this. Wikipedia has a topic on this:
Out of memory (OOM) is an often undesired state of computer operation where no additional memory can be allocated for use by programs or the operating system. Such a system will be unable to load any additional programs, and since many programs may load additional data into memory during execution, these will cease to function correctly. This occurs because all available memory, including disk swap space, has been allocated.
In general there should be a notice in either /var/log/syslog (debian based systems) or /var/log/messages. A grep "Killed process" /var/log/syslog should show if the system killed a process.
Interesting topics: "How to discover what is killing a process?" on stack and this article on lwn.net.
The project page for OOM has a lot of tips:
What causes these OOM events?
The kernel is really out of memory. The workload used more memory than the system has RAM and swap space. What are SwapFree and MemFree in /proc/meminfo? If both are very low (less than 1% of their total), then the workload may be at fault. (unless mlock() or HugeTLB are involved, see below...)
The kernel is out of low memory on 32-bit architectures. What is LowFree in /proc/meminfo? If it is very low, but HighFree is much higher, then you have this condition. This workload may benefit from being run on a 64-bit platform or kernel.
There is a kernel data structure or memory leak. What are SwapFree and MemFree in /proc/meminfo? What is the number of task_struct objects in slabinfo? Was the system forking so many processes that it ran out of memory? What objects in /proc/slabinfo take up the most space? If one kind of object is taking up a vast portion of the system's total memory, that object may be responsible. Check with the subsystem experts for the area from which that object comes. To see the object usage, run this on the command-line:
awk '{printf "%5d MB %s\n", $3*$4/(1024*1024), $1}' < /proc/slabinfo | sort -n
The kernel is not using its swap space properly. If the application uses mlock() or HugeTLBfs pages, it may not be able to use its swap space for that application. If this happens, SwapFree may still have a very large value when the OOM occurs. These two features do not allow the system to swap the affected memory out, however, so overusing them may exhaust system memory and leave the system with no other recourse. It is also possible for the system to find itself in a sort of deadlock. Writing data out to disk may, itself, require allocating memory for various I/O data structures. If the system cannot find even that memory, the very functions used to create free memory will be hamstrung and the system will likely run out of memory. It is possible to do some minor tuning to start paging earlier, but if the system cannot write dirty pages out fast enough to free memory, one can only conclude that the workload is mis-sized for the installed memory and there is little to be done. Raising the value in /proc/sys/vm/min_free_kbytes will cause the system to start reclaiming memory at an earlier time than it would have before. This makes it harder to get into these kinds of deadlocks. If you get these deadlocks, this is a good value to tune. If you run into a case where tuning this value helps, please report it. We may need to make changes to the default values.
The kernel has made a bad decision and mis-read its statistics. It went OOM while it still had plenty of good RAM to use.
Something really pathological is happening The kernel actually decides to go OOM after it has spend a "significant" amount of time scanning memory for something to free. As of 2.6.19, this "significant amount" happens after the VM has scanned an amount equal to all of the (currently) active+inactive pages in a zone six times. If the kernel is rapidly scanning pages, but your I/O devices (swap, filesystem, or network fs) are too slow, the kernel may judge that there is no progress being made and trigger an OOM even if there is swap free.
Run this script during your test, and the OOM. Run the script, send the output to a VM expert. Have them parse it. Then come back and update this page. ;)
