38

Sometimes I work with huge dumps of data I want to keep in memory for processing. Sometimes I miscalculate the amount of memory my program will produce, or a debugger multiplies the memory usage by a factor that exceeds my available memory.

Whenever I start a memory-hungry process, this is what I'd expect from a sane operating system: try to eat all free memory, then ask some other non-essential processes nicely to give up some memory they don't need, then write to swap.

Here's what Ubuntu does for me: eat all fre memory, then ask the operating system to swap all essential services (gnome session, terminal, keyboard), then freeze and wait for me to pull the power plug.

Two questions:

  1. How can an operating system assume, that anything could be so important that it is ok to stop listening to user input?
  2. How can I tell Ubuntu to never swap essential services and always react to user input, even if some stupid process tries to eat up more resources than the system provides.
15
  • How much RAM do you have installed? What size is your swap (in terminal, type swapon to find out)? Cheers, Al
    – heynnema
    Sep 1, 2016 at 19:01
  • 4
    16gb ram and 16gb swap. But that's not the point here, this problem cannot be solved by adding more memory.
    – Klamann
    Sep 5, 2016 at 11:18
  • 1
    Try one of two things. 1) change the swappiness setting to 10, ie: vm.swappiness = 10 in /etc/sysctl.conf. Search here for swappiness for more info about it. 2) If swappiness doesn't help... Even though you may not want to... increase the size of your swapfile to 1.5x16G and see if that helps. Keep me posted. Cheers, Al
    – heynnema
    Sep 5, 2016 at 14:33
  • 1
    @Klamann I agree adding more swap won't fix the problem. Once you have a broken program consuming all RAM+SWAP adding extra SWAP only delays the inevitable. Sep 5, 2016 at 19:57
  • 1
    @WinEunuuchs2Unix, as I said, vm.swappiness=10 needs to be ADDED to sysctl.conf. An experienced person could even use the sysctl command on the fly, to set vm.swappiness=10, without editing the sysctl.conf file. Cheers, Al ps: waiting for the OP to respond.
    – heynnema
    Sep 5, 2016 at 20:45

4 Answers 4

7

I still don't have a solution for the problem, but I can offer two workarounds which might be of interest to others:

1) earlyoom

That's a service that watches memory usage and kills the process that consumes the most memory when a certain threshold is reached (see also this and this question regarding the OOM killer in the linux kernel)

I've tested it with a demo process that indefinetly requests memory in small chunks. Here's my first impression: When I start the rogue process, it quickly eats up all of my RAM. Then swapping begins, the system becomes inresponsive. A few seconds later the system is back online. The log of earlyoom shows that it killed the memory eating process after both memory and swap usage reached 90%.

There is still the annoying lag when swapping begins and after the process was killed, some portions of other processes usually remain in swap until they are requested, but it's a start.

2) just disable swap

I know this is a controversial topic, but for the purpose of desktop systems and especially development machines where it can happen from time to time that a process tries to eat up all your memory, it makes sense: Without swap, the OOM killer just works as intended. When you run out of memory, it finds the best process to kill and gets rid of it. No lag, no delay.

You can disable swap for the current session with sudo swapoff -a or make the change permanent.


The proper solution for the problem would of course be that the system stays responsive when the main memory is depleted and it starts to swap memory like there's no tomorrow, but that doesn't seem to be happening any time soon.

1
  • 4
    I have disabled swap and my system goes straight from low memory (<100MB) to freezing. How can I tell if the OOM killer is actuall enabled?
    – Michael
    Apr 30, 2019 at 3:06
1

From +22.04 LTS

There's a service called systemd-oomd that will automatically monitor memory usage and attempt to kill processes when out of memory and it is installed by default as a part of systemd. To view the current status type oomctl. Example output:

Dry Run: no
Swap Used Limit: 90.00%
Default Memory Pressure Limit: 60.00%
Default Memory Pressure Duration: 20s
System Context:
        Memory: Used: 4.1G Total: 15.5G
        Swap: Used: 1.0G Total: 3.9G
Swap Monitored CGroups:
        Path: /
                Swap Usage: (see System Context)
Memory Pressure Monitored CGroups:
        Path: /user.slice/user-1000.slice/user@1000.service
                Memory Pressure Limit: 50.00%
                Pressure: Avg10: 0.00 Avg60: 0.02 Avg300: 0.00 Total: 9s
                Current Memory Usage: 3.9G
                Memory Min: 0B
                Memory Low: 0B
                Pgscan: 23663084
                Last Pgscan: 23663084

By default, start reclaiming at 90% usage of both memory and swap. It uses memory pressure on user slices control groups (i.e. user@1000.service) which is where most application spawn, but not on system.slice control group, which spawns the most crucial components of the system such as NetworkManager and Gnome Display Manager. In short, oomd will free user processes before system ones. To view the current configuration of systemd-oomd, type:

systemd-analyze cat-config systemd/oomd.conf

Refer to this page for a reference on what they do. To edit the OOM parameters of a specific systemd unit, add or edit the variables specified here in the unit file.

0

Try one of two things:

1) change the swappiness setting from its default setting of 60, to 10, ie: add vm.swappiness = 10 to /etc/sysctl.conf (in terminal, type sudo gedit /etc/sysctl.conf), then reboot the system. Search here for swappiness for more info about it.

2) If swappiness doesn't help... even though you may not want to... increase the size of your swapfile to 1.5x16G and see if that helps.

Keep me posted. Cheers, Al

10
  • I've set up a VM to run some tests, because rebooting my OS every few seconds got really annoying. Ubuntu 16.04, 2gb ram, 3gb swap, 20gb disk. Then I ran a script that eats up a lot of memory: With default swappiness (60), the system freezes and after a few minutes, I shut it down because the time to recovery was inacceptible. With swappiness 10, the system freezes for a few seconds, then it accepts input but you can't start any processes (e.g. top to kill the memory hog). After a minute or so, the process gets killed. Not perfect, but we're getting closer.
    – Klamann
    Sep 7, 2016 at 14:27
  • Keep us posted. The VM won't truly emulate your real life working OS, but it'll let you play with the settings. I'll be curious if swappiness helps with your problem. Cheers, Al
    – heynnema
    Sep 7, 2016 at 14:30
  • Ah, good. Progress! Read up a little on swappiness. You can play with the value a little bit. Cheers, Al
    – heynnema
    Sep 7, 2016 at 14:32
  • How much swap was being used when the system froze? Cheers, Al
    – heynnema
    Sep 7, 2016 at 14:35
  • 2
    why, what's the point in that? more memory wont prevent the system from getting stuck if I suck up that memory too.
    – Klamann
    Sep 7, 2016 at 15:00
0

I solved a similar problem. I don't know if my experience maybe suitable to you...

Recently I have published a guide on how to install linux on loopback LVM devices booting from USB (so without have to install grub on the internal disk, leaving it as original). Here is the guide: https://github.com/DareDevil73/linux-on-loopback-usb .

Then I fallen in the freeze issue on high memory load and I observed an abnormal swap space usage (all the RAM eaten, and swap usage near to zero). Obviously the LVM swap partition was mounted and working properly, but I don't known why the kernel did not use it as expected.

I tried an alternative solution. I have created a swap loopback file (not LVM) and the freeze is gone. Now the swap file is used as it would be, and the OS never freeze!

Please look at https://github.com/DareDevil73/linux-on-loopback-usb#known-issues to get deeper information.

1
  • Please expand the links into answers. Oct 23, 2019 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.