4

Everything is in the title. I'm running a play framework and a redis process on my ubuntu 2GB RAM VPS but last night the 2 processes suddenly crashed without any log (which is weird because they always write logs because crashing).

Thus, I am wondering if Ubuntu could be the cause of these crashes? The memory was very low. Could Ubuntu kill these processes to free the memory?

If yes, is there any log somewhere that I can read to check what Ubuntu did, why and when exactly? Also, is there a way to disable these "auto kill" or tell ubuntu to kill other processes but not these 2 processes?

4
  • @Rinzwind thank you for your links it was very helpful forme to understand! Could you also tell me if the OOM killer writes logs somewhere? (so that I can know exactly which processes have been killed and when). And also, I know that a signal is sent to the process when a user presses CTRL+C for example. Is there such a signal I can intercept when the OOM Killer wants to kill my process? Apr 2, 2015 at 7:50
  • @Rinzwind Thank you But i can't see anything about killed processes in the log. Does it mean my processes have not been killed by the kernel? Apr 2, 2015 at 12:45
  • @Rinzwind Ok thank you. I guess you can create an answer with your comments. They helped me and all answers are in the links Apr 2, 2015 at 17:00
  • done :) added as much as I could find.
    – Rinzwind
    Apr 3, 2015 at 9:22

1 Answer 1

3

It is not Ubuntu that does this. It is the Linux kernel that does this. Wikipedia has a topic on this:

Out of memory (OOM) is an often undesired state of computer operation where no additional memory can be allocated for use by programs or the operating system. Such a system will be unable to load any additional programs, and since many programs may load additional data into memory during execution, these will cease to function correctly. This occurs because all available memory, including disk swap space, has been allocated.

In general there should be a notice in either /var/log/syslog (debian based systems) or /var/log/messages. A grep "Killed process" /var/log/syslog should show if the system killed a process.

Interesting topics: "How to discover what is killing a process?" on stack and this article on lwn.net.


The project page for OOM has a lot of tips:

What causes these OOM events?

  • The kernel is really out of memory. The workload used more memory than the system has RAM and swap space. What are SwapFree and MemFree in /proc/meminfo? If both are very low (less than 1% of their total), then the workload may be at fault. (unless mlock() or HugeTLB are involved, see below...)

  • The kernel is out of low memory on 32-bit architectures. What is LowFree in /proc/meminfo? If it is very low, but HighFree is much higher, then you have this condition. This workload may benefit from being run on a 64-bit platform or kernel.

  • There is a kernel data structure or memory leak. What are SwapFree and MemFree in /proc/meminfo? What is the number of task_struct objects in slabinfo? Was the system forking so many processes that it ran out of memory? What objects in /proc/slabinfo take up the most space? If one kind of object is taking up a vast portion of the system's total memory, that object may be responsible. Check with the subsystem experts for the area from which that object comes. To see the object usage, run this on the command-line:

    awk '{printf "%5d MB %s\n", $3*$4/(1024*1024), $1}' < /proc/slabinfo | sort -n
    
  • The kernel is not using its swap space properly. If the application uses mlock() or HugeTLBfs pages, it may not be able to use its swap space for that application. If this happens, SwapFree may still have a very large value when the OOM occurs. These two features do not allow the system to swap the affected memory out, however, so overusing them may exhaust system memory and leave the system with no other recourse. It is also possible for the system to find itself in a sort of deadlock. Writing data out to disk may, itself, require allocating memory for various I/O data structures. If the system cannot find even that memory, the very functions used to create free memory will be hamstrung and the system will likely run out of memory. It is possible to do some minor tuning to start paging earlier, but if the system cannot write dirty pages out fast enough to free memory, one can only conclude that the workload is mis-sized for the installed memory and there is little to be done. Raising the value in /proc/sys/vm/min_free_kbytes will cause the system to start reclaiming memory at an earlier time than it would have before. This makes it harder to get into these kinds of deadlocks. If you get these deadlocks, this is a good value to tune. If you run into a case where tuning this value helps, please report it. We may need to make changes to the default values.

  • The kernel has made a bad decision and mis-read its statistics. It went OOM while it still had plenty of good RAM to use.

  • Something really pathological is happening The kernel actually decides to go OOM after it has spend a "significant" amount of time scanning memory for something to free. As of 2.6.19, this "significant amount" happens after the VM has scanned an amount equal to all of the (currently) active+inactive pages in a zone six times. If the kernel is rapidly scanning pages, but your I/O devices (swap, filesystem, or network fs) are too slow, the kernel may judge that there is no progress being made and trigger an OOM even if there is swap free.

Run this script during your test, and the OOM. Run the script, send the output to a VM expert. Have them parse it. Then come back and update this page. ;)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.