39

Ubuntu 22.04 comes with the systemd-oomd service enabled by default, which has been "helpfully" killing my IDE and / or terminals whenever I try to compile an application using an abundance of threads / memory.

What is the right way to either turn this off, or configure the service to not shoot random processes in the face while I'm using them?

I'm aware that I can mitigate this behavior in a few ways; e.g. by increasing the size of the swap space, but this is still not a panacea since:

  • The OOM daemon kills the entire process tree, so even the terminal hosting the processes that were killed will suddenly vanish;

  • The OOM daemon kills the process tree without providing any notification to the user, so all the user knows is that their terminal / IDE / application hosting memory-hungry processes has suddenly vanished.

A user could find out what happened post-hoc via journalctl or something similar if they knew what to look for, but I don't think the average Ubuntu desktop user would think to do this.

As an example, normally when a process crashes via a deadly signal or similar, a crash reporter will tell the user that something went wrong. Shouldn't there be a similar facility for processes killed by the OOM daemon?


Edited to add requested output re: swap space; as far as I know these are just the defaults that were set when Ubuntu 22.04 was installed.

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            31Gi       5.2Gi       3.1Gi       210Mi        23Gi        25Gi
Swap:          2.0Gi       0.0Ki       2.0Gi

$ sysctl vm.swappiness
vm.swappiness = 60

$ swapon -s
Filename                Type        Size        Used        Priority
/swapfile                               file        2097148     792     -2
2
  • Comments are not for extended discussion; this conversation has been moved to chat.
    – andrew.46
    Aug 16, 2022 at 9:32
  • For me this didn't help. I have 16GB of ram and had only 1GB of swap, which was constantly full. Now I increased swap file size to 16GB and so far - so good. There are a lot of pages recommending optimal size of swap file so if this doesn't help check on those.
    – MilanG
    Jan 23, 2023 at 9:03

5 Answers 5

32

Most systemd services can be managed via the systemctl utility. In this case, we want to disable the systemd-oomd service. This can be done with:

$ systemctl disable --now systemd-oomd

You should see something like (depending on your OS):

$ systemctl disable --now systemd-oomd
Removed /etc/systemd/system/multi-user.target.wants/systemd-oomd.service.
Removed /etc/systemd/system/dbus-org.freedesktop.oom1.service.

You can then verify that the service is disabled, with:

$ systemctl is-enabled systemd-oomd

And you should then see:

$ systemctl is-enabled systemd-oomd
disabled

It is possible, however, that other services might attempt to restart the systemd-oomd service. To prevent this, you can 'mask' the service. For example:

$ systemctl mask systemd-oomd
Created symlink /etc/systemd/system/systemd-oomd.service → /dev/null.

And then systemctl is-enabled should now report:

$ systemctl is-enabled systemd-oomd
masked

If you'd like to later unmask (or re-enable) a service, that can be done with, for example,

$ systemctl enable systemd-oomd
$ systemctl unmask systemd-oomd

See man systemctl for more details; in particular, note the caveats regarding masking of systemd services.

2
  • Could you add something in the answer about how to reverse the masking. Thanks. Mar 22, 2023 at 14:56
  • 2
    The solution for me was finally uninstalling systemd-oomd. Disabling was not enough and masking had annoying messages. You can simply reinstall it later if you want it back.
    – daltux
    Aug 18, 2023 at 12:05
9

Just to point out, systemd-oomd doesn't just kill on oom (out of memory and swap) -- if it did, that'd probably be fine. It (either also or exclusively, not sure which..) kills based on what it deems excessive swap activity (the message I found on mine indicates Killed (gnome-terminal's scope) "due to memory pressure for " (the slice) "being 58.97% > 50.00% for >20s with reclaim activity."

I (this is the same in Ubuntu 20.04 and 22.04) run some junk all day while I'm away, it gradually swaps out bits of the copy of firefox that's open, when I get home, go to use firefox at which point it has to swap some of it back in. Which is fine, but system-oomd (this has happened to me twice within a few days after upgrade to Ubuntu 22.04) decided it's not OK and killed my gnome-terminal with a screen session with like 8 or 9 shells, VirtualBox, some s3ql mounts, etc. all running under it. I was really afraid something in my upgrade had gone wrong and gnome-terminal and such were crashing on their own, glad to see that's not the case. Thanks systemd! Anyway, at least it's easy to disable!

Edit: FYI, I'm able to run:

sudo apt remove systemd-oomd

as well. It's apparently just installed directly, not as a dependency of the systemd package or anything like that.

3

With 32G RAM, the default 2G /swapfile would normally be enough. However if you're running some large memory-hungry apps, or have large data-sets, it may need to swap more, so we'll increase it from 2G to 4G and see if that helps. Don't disable OOM.

Note: Incorrect use of the rm and dd commands can cause data loss. Suggest copy/paste.

In the terminal...

sudo swapoff -a           # turn off swap
sudo rm -i /swapfile      # remove old /swapfile

sudo dd if=/dev/zero of=/swapfile bs=1M count=4096

sudo chmod 600 /swapfile  # set proper file protections
sudo mkswap /swapfile     # init /swapfile
sudo swapon /swapfile     # turn on swap
free -h                   # confirm 32G RAM and 4G swap

Edit /etc/fstab, using sudo -H gedit /etc/fstab or sudo pico /etc/fstab.

Confirm this /swapfile line in /etc/fstab... and confirm no other “swap” lines... use SPACES in this line... confirm NO TABS...

/swapfile  none  swap  sw  0  0

reboot                    # reboot and verify operation
7
  • 3
    This is a 32GB system that is running out of memory. I have 8GB and never experienced a crash in years before upgrading to ubuntu 22.04. There is definitely a problem with the systemd-oomd or with the configuration. I doubt that increasing the swap from 2GB to 4GB would be enough. In the old days, the rule of thumb was that the swap space should be the same as the ram space. I'm considering buying 32GB ram, but this example where crashes occurs even with a 32GB PC is worrisome.
    – chmike
    May 6, 2022 at 5:20
  • @chmike It really depends on the user's work set. If they're using a lot of memory-hungry applications, or working with large data sets, then RAM/swap settings can make a difference. Since this user didn't try my answer, and see if it was enough to solve their problem, we'll never know. They decided to cripple their system.
    – heynnema
    May 6, 2022 at 13:10
  • 2
    I would say that it is systemd-oomd that cripples the system and will give a bad experience to new Ubuntu users. How come I see crashes with Ubuntu 22.04 and never saw any with previous versions by doing the exact same activity (same number of apps, browser tabs, etc.). There is definitely something fishy with this systemd service. Beside, simply killing apps without any visual feedback (dialog) explaining to the user what happen and why is incredibly bad UI design. Note that the user may loose data in this process. Systemd-oomd is crippling Ubuntu
    – chmike
    May 7, 2022 at 7:33
  • 2
    If a user with 32GB of ram has apps killed because there is not enough memory, I seriously doubt that adding 2GB of swap space will definitely fix the problem.
    – chmike
    May 9, 2022 at 8:29
  • 1
    @chmike You may be right. However, a 2G /swapfile is default, and I've seen too many cases where 2G just wasn't enough. But... we'll never know if 4G would have helped, as the user has chosen another route.
    – heynnema
    May 9, 2022 at 9:22
0

I had a very similar problem as discribed by hwertz. Firefox caused a crash of the whole system.

Since neither increasing the swap nor disabling/masking the systemd-oomd service did the job on my system, I want to add my solution here.

In my case it is a rather old system with an NVIDIA Geforce 9500 GT. The card requires the 340 driver which is not included anymore in Ubuntu 22.04. Already after the upgrade from 20.04 I had problems (no login screen,..) and I had to remove all NVIDIA* packages via

sudo apt purge nvidia*

Well using nouveau solved the login problem, but triggered the Firefox freeze. Finally the problem's solution on my old system was to get a working 340-driver version for Ubuntu 22.04.

Note: I am not sure if this would also happen with a fresh installation. Can be that there is a misconfiguration of the system due to the upgrade from 20.04

0

Bit of a necro. You can increase swap size but, changing swap size to 4gb from default 2gb does nothing for me. oomd.conf can be edited so you can try that. My drive is small so I don't want to increase swap size any further than 8gb and I don't want to turn swap off either. As far as I know, oomd is very important for not crashing your whole pc and corrupting everything from running out of memory. So I don't want it turned off completely.

systemd-analyze cat-config systemd/oomd.conf

https://man.archlinux.org/man/oomd.conf.5.en

https://man.archlinux.org/man/systemd-oomd.service.8.en

Also I have no idea what I'm doing so do this at your own risk. haha. It's read-only for a reason I guess.

Since it is read-only you can make a file and then edit it by doing this

sudo touch [filename]
sudo nano [filename]

I've changed duration to 60s and pressure to 90%. Also swap limit to 95%. Also for testing you can add the flag --dry-run to systemctl stuff. Instead of doing what it would do it prints what it would do to a log.

sudo systemctl --dry-run start systemd-oomd

EDIT:

new solution: delete this trash

sudo apt remove systemd-oomd

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .