2

I have some swap configured and I believe activated...

Filename                Type        Size    Used    Priority
/dev/sda5                               partition   7811068 1124912 200
/mnt/data02/swapfile                    file        134217724   37032   100
/home/swapfile                          file        134217724   36600   -1

But, when the system monitor shows memory reaching 100% the system tends to respond by closing/crashing programs. Xorg has crashed in this way as has the wireless driver. While this happens the system monitor shows little (under 5 GiB) use of the Swap. I have confirmed that swappiness is not set to an extreme value (and changes in this parameter have appeared to have no effect on the issue).

~$  cat /proc/sys/vm/swappiness
70

The system has a huge amount of RAM...

~$ free -h
             total       used       free     shared    buffers     cached
Mem:          125G        20G       105G       161M        54M       1.1G
-/+ buffers/cache:        19G       106G
Swap:         263G       1.1G       262G

... but sometimes I run overbudget on Memory resources and it would be nice if it failed more gracefully than things crashing.

What can I do to resolve this issue?

EDIT

~$ cat /etc/fstab
    # <file system> <mount point>   <type>  <options>       <dump>  <pass>
    # / was on /dev/sda3 during installation
    UUID=8dfbed62-9957-4f06-b4e1-a42020adec91 /               ext4    errors=remount-ro 0       1
    # /home was on /dev/sda6 during installation
    UUID=b6f33408-1d8b-4302-9983-5c778ef64f47 /home           ext4    defaults        0       2
    # swap was on /dev/sda5 during installation
    # ae0304dd-e63e-4d3a-99da-9c9d7a034c6e is the swap file
    UUID=fd4c00c9-49bf-4562-adea-1c817fc57ce9 none            swap    sw,pri=200              0       0
    UUID=3A323DCA323D8BBF /mnt/data01 ntfs-3g defaults,windows_names,locale=en_US.utf8  0 0
    UUID=4cc8a19d-5991-4186-8f65-7062805b66a6 /mnt/data02 ext4 defaults 0 0
    /mnt/data02/swapfile   none    swap    sw,pri=100    0   0
    /home/swapfile  none  swap  sw  0,pri=150 0

EDIT 2 In response to a comment below I watched my system perform an operation I knew would use all available RAM but not all available swap and then checked dmesg during and after the failure. The system swapped and became intermittently non-responsive (okay behavior). Then while swap was at less than 10% of capacity, Chrome crashed (Sorry, the program "chrome" closed unexpectedly. Your computer does not have enough free memory to automatically analyze the problem and send a report to the developers). Trying to get back to the dmesg output I was watching I got an error message that stated This window is not responding. Do you want to force the application to exit, or wait for it to respond. I selected 'wait'. The desktop reappeared and the gnome system monitor went in and out of being greyed out several times while the system swapped. When I checked back in, I was at the Ubuntu login screen. I logged in as normal... all of my earlier running processes were gone and I was greeted with an error message identical to the one about Chrome in reference to Xorg. Checking dmesg shows only the following two messages:

[131267.206774] Watchdog[3433]: segfault at 0 ip 00007fe38faf9756 sp 00007fe37f393770 error 6 in chrome[7fe38be0a000+510c000]
[133329.875212] nvidia 0000:03:00.0: irq 106 for MSI/MSI-X

EDIT 3 Other possible relevant topics:

Edit 4 Additional error messages:

[92315.165728] Watchdog[1319]: segfault at 0 ip 00007f7d0a417756 sp 00007f7cf9cb1770 error 6 in chrome[7f7d06728000+510c000]
[92656.478271] INFO: task Chrome_IOThread:1292 blocked for more than 120 seconds.
[92656.478275]       Tainted: P           OX 3.13.0-45-generic #74-Ubuntu
[92656.478276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[92656.478278] Chrome_IOThread D ffff88207fc534c0     0  1292  32756 0x00000000
[92656.478282]  ffff881fa9d15dd8 0000000000000086 ffff881fad2ab000 ffff881fa9d15fd8
[92656.478285]  00000000000134c0 00000000000134c0 ffff881fad2ab000 ffff881fad2ab000
[92656.478288]  ffff881f5fce6260 ffff881f5fce6268 ffffffff00000000 ffff881f5fce6270
[92656.478290] Call Trace:
[92656.478299]  [<ffffffff817252d9>] schedule+0x29/0x70
[92656.478302]  [<ffffffff81727f55>] rwsem_down_write_failed+0x115/0x230
[92656.478307]  [<ffffffff81371d63>] call_rwsem_down_write_failed+0x13/0x20
[92656.478311]  [<ffffffff81314c90>] ? apparmor_file_mprotect+0x30/0x30
[92656.478313]  [<ffffffff8172796d>] ? down_write+0x2d/0x30
[92656.478318]  [<ffffffff8116ba7c>] vm_mmap_pgoff+0x6c/0xc0
[92656.478322]  [<ffffffff8117f916>] SyS_mmap_pgoff+0x116/0x270
[92656.478325]  [<ffffffff81018802>] SyS_mmap+0x22/0x30
[92656.478328]  [<ffffffff8173196d>] system_call_fastpath+0x1a/0x1f
16
  • Er... s/complaint/compliant. Feb 12, 2015 at 18:53
  • Try increasing swappiness.
    – XperianX
    Feb 12, 2015 at 20:22
  • Already tried swappiness @ 90 and there was no improvement. Feb 12, 2015 at 20:27
  • 2
    Mentioned here: productforums.google.com/forum/#!topic/chrome/zjjitIYfuxw is checking if it is the graph.card with "/usr/bin/google-chrome-stable --disable-gpu %U"
    – Rinzwind
    Feb 15, 2015 at 16:00
  • 1
    Not sure about this, but privileged processes can ask allocating memory (with mmap(...,MAP_LOCKED)) just in RAM, non-swappable, etc. Both chrome and the graphic drivers do have suid parts (look at chrome sandbox). So you can have an OOM condition even with swap available --- just for this kind of memory. I do not know if this can trigger the OOM killer, or just the allocation fails and the process kill itself -- this is why this is just a comment and not an answer.
    – Rmano
    Feb 19, 2015 at 8:29

2 Answers 2

3
+50

Depending on what they are doing, Computers with large amounts of memory, such as yours, can get into difficulties when the amount of free memory (resident as opposed to swap) becomes very low. Sometimes (not sure for your situation) things can be improved by increasing the minimum amount of memory kept free, or /proc/sys/vm/min_free_kbytes. Think of it as keeping more room free such that things are easier to move around and re-group and de-fragment and such. Try a very large number first, say 20G, and if that helps try to reduce it. You might also help yourself by watching "free" carefully in an attempt to correlate the issues with some minimum available memory.

Method 1 (script run as sudo):

#! /bin/bash
cat /proc/sys/vm/min_free_kbytes

echo "20000000" > /proc/sys/vm/min_free_kbytes

cat /proc/sys/vm/min_free_kbytes

Method 2 (direct command):

echo "20000000" | sudo tee /proc/sys/vm/min_free_kbytes
5
  • Do you have a specific thought/reasoning behind the 20 GB number? I'll give it a shot and respond back (it was set at 67584 kb). Feb 15, 2015 at 5:03
  • I just wanted to suggest a very large number, something that should be overkill, as a starting test. No use wasting a lot of time, if this isn't the solution. A few months ago I helped on an Ubuntu forums thread, where this solution did fix the issue. I think that one ended up at 10G min_free out of about 380G. Feb 15, 2015 at 6:57
  • Doug, your solution helped me get a more informative error message and left the system more responsive during early parts of an out of RAM (but swap left over) scenario. However, after trying various combinations the note @Rinzwind left about a Chrome bug when the GPU is enabled appears to be the prime culprit. Would you care to incorporate it into your answer? Alternatively Rinzwind, would you care to post your comment as an answer? Feb 16, 2015 at 14:25
  • I'll get back to you on this, but am out of town for a day or two. Feb 16, 2015 at 20:57
  • 1
    I agree that what I suggested helped by giving a bit more time which gave better errors messages leading to the more helpful suggestion. However, the amount of memory you have is much more than most typical users, and thus the extra time simply might not be made available for most systems. I am reluctant to edit my answer to include the @Rinzwind note for two reasons: First, it was not my insight; Second, and as explained above, I don't know that extra insight for this particular type of issue would be a gained for the general user. Feb 18, 2015 at 21:53
1

To find out if your processes have been killed by the OOM killer you can inspect the result of this command:

sudo egrep -ri 'killed process' /var/log/ | grep -v auth.log

If this is the case you might want to look at the article about taiming the OOM Killer. http://lwn.net/Articles/317814/

1
  • Thanks, it doesn't seem like it was the OOM killer in my case because the command you mention above doesn't yield any resulting lines. However, it does seem like a very useful piece of knowledge to have in response to this question. Feb 20, 2015 at 11:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.