10

Ubuntu 22.04 comes with the systemd-oomd service enabled by default, which has been "helpfully" killing my IDE and / or terminals whenever I try to compile an application using an abundance of threads / memory.

What is the right way to either turn this off, or configure the service to not shoot random processes in the face while I'm using them?

I'm aware that I can mitigate this behavior in a few ways; e.g. by increasing the size of the swap space, but this is still not a panacea since:

  • The OOM daemon kills the entire process tree, so even the terminal hosting the processes that were killed will suddenly vanish;

  • The OOM daemon kills the process tree without providing any notification to the user, so all the user knows is that their terminal / IDE / application hosting memory-hungry processes has suddenly vanished.

A user could find out what happened post-hoc via journalctl or something similar if they knew what to look for, but I don't think the average Ubuntu desktop user would think to do this.

As an example, normally when a process crashes via a deadly signal or similar, a crash reporter will tell the user that something went wrong. Shouldn't there be a similar facility for processes killed by the OOM daemon?


Edited to add requested output re: swap space; as far as I know these are just the defaults that were set when Ubuntu 22.04 was installed.

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            31Gi       5.2Gi       3.1Gi       210Mi        23Gi        25Gi
Swap:          2.0Gi       0.0Ki       2.0Gi

$ sysctl vm.swappiness
vm.swappiness = 60

$ swapon -s
Filename                Type        Size        Used        Priority
/swapfile                               file        2097148     792     -2
18
  • How much swap do you have? If you have very little, or worse, none, that could cause that issue. Swap can be used as extra RAM for when you run out. If you have none, then manually create a swap file. 8GB is a reasonable size to start with, although if you like to have tons of apps open, you could go up to 16 GB. It really isn't advisable to disable systemd-oomd (is it even possible?).
    – cocomac
    Apr 27 at 0:23
  • 2
    @cocomac re: "Swap can be used as extra RAM for when you run out.". This part of your comment isn't entirely accurate. Swap moves "least recently used pages of memory" in/out of a swapfile or swap partition, typically when RAM is tight.
    – heynnema
    Apr 27 at 1:24
  • Edit your question and show me free -h and sysctl vm.swappiness and swapon -s. Start comments to me with @heynnema or I'll miss them.
    – heynnema
    Apr 27 at 1:25
  • 1
    @KevinUshey Did you increase your swap as per my answer?
    – heynnema
    Apr 30 at 23:49
  • 1
    @KevinUshey Go to memtest86.com and download/run their free memtest to test your memory. Get at least one complete pass of all the 4/4 tests to confirm good memory. This may take a few hours to complete.
    – heynnema
    Apr 30 at 23:50

2 Answers 2

13

Most systemd services can be managed via the systemctl utility. In this case, we want to disable the systemd-oomd service. This can be done with:

$ systemctl disable --now systemd-oomd

You should see something like (depending on your OS):

$ systemctl disable --now systemd-oomd
Removed /etc/systemd/system/multi-user.target.wants/systemd-oomd.service.
Removed /etc/systemd/system/dbus-org.freedesktop.oom1.service.

You can then verify that the service is disabled, with:

$ systemctl is-enabled systemd-oomd

And you should then see:

$ systemctl is-enabled systemd-oomd
disabled

It is possible, however, that other services might attempt to restart the systemd-oomd service. To prevent this, you can 'mask' the service. For example:

$ systemctl mask systemd-oomd
Created symlink /etc/systemd/system/systemd-oomd.service → /dev/null.

And then systemctl is-enabled should now report:

$ systemctl is-enabled systemd-oomd
masked

See man systemctl for more details; in particular, note the caveats regarding masking of systemd services.

2

With 32G RAM, the default 2G /swapfile would normally be enough. However if you're running some large memory-hungry apps, or have large data-sets, it may need to swap more, so we'll increase it from 2G to 4G and see if that helps. Don't disable OOM.

Note: Incorrect use of the rm and dd commands can cause data loss. Suggest copy/paste.

In the terminal...

sudo swapoff -a           # turn off swap
sudo rm -i /swapfile      # remove old /swapfile

sudo dd if=/dev/zero of=/swapfile bs=1M count=4096

sudo chmod 600 /swapfile  # set proper file protections
sudo mkswap /swapfile     # init /swapfile
sudo swapon /swapfile     # turn on swap
free -h                   # confirm 32G RAM and 4G swap

Edit /etc/fstab, using sudo -H gedit /etc/fstab or sudo pico /etc/fstab.

Confirm this /swapfile line in /etc/fstab... and confirm no other “swap” lines... use SPACES in this line... confirm NO TABS...

/swapfile  none  swap  sw  0  0

reboot                    # reboot and verify operation
7
  • 1
    This is a 32GB system that is running out of memory. I have 8GB and never experienced a crash in years before upgrading to ubuntu 22.04. There is definitely a problem with the systemd-oomd or with the configuration. I doubt that increasing the swap from 2GB to 4GB would be enough. In the old days, the rule of thumb was that the swap space should be the same as the ram space. I'm considering buying 32GB ram, but this example where crashes occurs even with a 32GB PC is worrisome.
    – chmike
    May 6 at 5:20
  • @chmike It really depends on the user's work set. If they're using a lot of memory-hungry applications, or working with large data sets, then RAM/swap settings can make a difference. Since this user didn't try my answer, and see if it was enough to solve their problem, we'll never know. They decided to cripple their system.
    – heynnema
    May 6 at 13:10
  • 1
    I would say that it is systemd-oomd that cripples the system and will give a bad experience to new Ubuntu users. How come I see crashes with Ubuntu 22.04 and never saw any with previous versions by doing the exact same activity (same number of apps, browser tabs, etc.). There is definitely something fishy with this systemd service. Beside, simply killing apps without any visual feedback (dialog) explaining to the user what happen and why is incredibly bad UI design. Note that the user may loose data in this process. Systemd-oomd is crippling Ubuntu
    – chmike
    May 7 at 7:33
  • 1
    @chmike No reason to down-vote my answer. I can't fix 22.04.
    – heynnema
    May 7 at 11:51
  • 1
    If a user with 32GB of ram has apps killed because there is not enough memory, I seriously doubt that adding 2GB of swap space will definitely fix the problem.
    – chmike
    May 9 at 8:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.