22

Sometimes I work with huge dumps of data I want to keep in memory for processing. Sometimes I miscalculate the amount of memory my program will produce, or a debugger multiplies the memory usage by a factor that exceeds my available memory.

Whenever I start a memory-hungry process, this is what I'd expect from a sane operating system: try to eat all free memory, then ask some other non-essential processes nicely to give up some memory they don't need, then write to swap.

Here's what Ubuntu does for me: eat all fre memory, then ask the operating system to swap all essential services (gnome session, terminal, keyboard), then freeze and wait for me to pull the power plug.

Two questions:

  1. How can an operating system assume, that anything could be so important that it is ok to stop listening to user input?
  2. How can I tell Ubuntu to never swap essential services and always react to user input, even if some stupid process tries to eat up more resources than the system provides.
  • How much RAM do you have installed? What size is your swap (in terminal, type swapon to find out)? Cheers, Al – heynnema Sep 1 '16 at 19:01
  • 3
    16gb ram and 16gb swap. But that's not the point here, this problem cannot be solved by adding more memory. – Klamann Sep 5 '16 at 11:18
  • Try one of two things. 1) change the swappiness setting to 10, ie: vm.swappiness = 10 in /etc/sysctl.conf. Search here for swappiness for more info about it. 2) If swappiness doesn't help... Even though you may not want to... increase the size of your swapfile to 1.5x16G and see if that helps. Keep me posted. Cheers, Al – heynnema Sep 5 '16 at 14:33
  • @heynnema On my Kernel 4.1.13 version of Ubuntu 14.04 /etc/sysctl.conf doesn't contain the vm.swappiness variable. When booting Kernel 4.7.2 under Ubuntu 16.04 the /etc/sysctl.conf file doesn't even exist and I think I read somewhere that systemd documentation says to use /etc/sysctl.d/ directory's configuration files instead of sysctl.conf. FYI. – WinEunuuchs2Unix Sep 5 '16 at 19:53
  • 1
    @Klamann I agree adding more swap won't fix the problem. Once you have a broken program consuming all RAM+SWAP adding extra SWAP only delays the inevitable. – WinEunuuchs2Unix Sep 5 '16 at 19:57
5

I still don't have a solution for the problem, but I can offer two workarounds which might be of interest to others:

1) earlyoom

That's a service that watches memory usage and kills the process that consumes the most memory when a certain threshold is reached (see also this and this question regarding the OOM killer in the linux kernel)

I've tested it with a demo process that indefinetly requests memory in small chunks. Here's my first impression: When I start the rogue process, it quickly eats up all of my RAM. Then swapping begins, the system becomes inresponsive. A few seconds later the system is back online. The log of earlyoom shows that it killed the memory eating process after both memory and swap usage reached 90%.

There is still the annoying lag when swapping begins and after the process was killed, some portions of other processes usually remain in swap until they are requested, but it's a start.

2) just disable swap

I know this is a controversial topic, but for the purpose of desktop systems and especially development machines where it can happen from time to time that a process tries to eat up all your memory, it makes sense: Without swap, the OOM killer just works as intended. When you run out of memory, it finds the best process to kill and gets rid of it. No lag, no delay.

You can disable swap for the current session with sudo swapoff -a or make the change permanent.


The proper solution for the problem would of course be that the system stays responsive when the main memory is depleted and it starts to swap memory like there's no tomorrow, but that doesn't seem to be happening any time soon.

  • 1
    I have disabled swap and my system goes straight from low memory (<100MB) to freezing. How can I tell if the OOM killer is actuall enabled? – Michael Apr 30 at 3:06
0

Try one of two things:

1) change the swappiness setting from its default setting of 60, to 10, ie: add vm.swappiness = 10 to /etc/sysctl.conf (in terminal, type sudo gedit /etc/sysctl.conf), then reboot the system. Search here for swappiness for more info about it.

2) If swappiness doesn't help... even though you may not want to... increase the size of your swapfile to 1.5x16G and see if that helps.

Keep me posted. Cheers, Al

  • I've set up a VM to run some tests, because rebooting my OS every few seconds got really annoying. Ubuntu 16.04, 2gb ram, 3gb swap, 20gb disk. Then I ran a script that eats up a lot of memory: With default swappiness (60), the system freezes and after a few minutes, I shut it down because the time to recovery was inacceptible. With swappiness 10, the system freezes for a few seconds, then it accepts input but you can't start any processes (e.g. top to kill the memory hog). After a minute or so, the process gets killed. Not perfect, but we're getting closer. – Klamann Sep 7 '16 at 14:27
  • Keep us posted. The VM won't truly emulate your real life working OS, but it'll let you play with the settings. I'll be curious if swappiness helps with your problem. Cheers, Al – heynnema Sep 7 '16 at 14:30
  • Ah, good. Progress! Read up a little on swappiness. You can play with the value a little bit. Cheers, Al – heynnema Sep 7 '16 at 14:32
  • How much swap was being used when the system froze? Cheers, Al – heynnema Sep 7 '16 at 14:35
  • 2
    why, what's the point in that? more memory wont prevent the system from getting stuck if I suck up that memory too. – Klamann Sep 7 '16 at 15:00
0

I solved a similar problem. I don't know if my experience maybe suitable to you...

Recently I have published a guide on how to install linux on loopback LVM devices booting from USB (so without have to install grub on the internal disk, leaving it as original). Here is the guide: https://github.com/DareDevil73/linux-on-loopback-usb .

Then I fallen in the freeze issue on high memory load and I observed an abnormal swap space usage (all the RAM eaten, and swap usage near to zero). Obviously the LVM swap partition was mounted and working properly, but I don't known why the kernel did not use it as expected.

I tried an alternative solution. I have created a swap loopback file (not LVM) and the freeze is gone. Now the swap file is used as it would be, and the OS never freeze!

Please look at https://github.com/DareDevil73/linux-on-loopback-usb#known-issues to get deeper information.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.