How do I use oom_score_adj?

Question

On 11.04, /proc/[pid] contains oom_score_adj, but man proc describes the old oom_adj. I have googled unsuccessfully for any information on how to use this.

What I need to arrange is that processes started by 'pbs' or 'gridengine' are killed before anything else gets killed. How can I do that?

Answer 1

^{Based on my own Q&A on Unix&Linux on a similar question.}

As Stuart pointed out very well in his answer, valid values are integers in the range of -1000 to 1000 for oom_score_adj. The lower the value, the lower the chance that it's going to be killed.

It's very inconvenient to have to change this value over and over again once you restart the application. The information is simply lost after the process has terminated. Upstart (the init daemon in Ubuntu til 2014, then moved to systemd), has a nice option for this to configure for daemons to make sure it's set whenever it has (re)started:

oom score

[...] snip [...]

Example:

# this application is a "resource hog"
oom score 1000

expect daemon
respawn
exec /usr/bin/leaky-app

So, basically, you can edit the /etc/init/myservice.conf configuration file for the services you like to change, to include a line oom score -1000. I do assume that the services 'pbs' or 'gridengine' you're talking about in your question is Upstart-enabled, otherwise you'll need another way to change this permanently.

Answer 2

If you can modify the start script spawning the process that you want to prioritize for oom-killer, add the following to the start script:

echo 1000 > /proc/self/oom_score_adj

The value is inherited for child processes.

Answer 3

We have choom now, from util-linux (display and adjust OOM-killer score):

choom --adjust -100 myprogram

The man page has more information.

Answer 4

If you set a high(ish) value for either oom_adj or oom_score_adj then they will be killed first. e.g.

echo 15 > /proc/[pid]/oom_adj

oom_adj goes from -16 to 15 and as mentioned above oom_score_adj accepts -1000 to 1000

Answer 5

This is an old question. With systemd in recent systems it couldn't be easier.

As you prefer, export EDITOR=vim or whatnot, then:

systemctl edit mysql.service

and input the two drop-in lines:

[Service]
OOMScoreAdjust=-150

save & exit; done — permanently.

---

Use systemctl edit --full if you're confused by the concept of config drop-in (just be careful to avoid writing to /usr/lib/systemd/system/*.service as your edits will get destroyed on updates).

---

Doc: man systemd.exec

OOMScoreAdjust=

Sets the adjustment level for the Out-Of-Memory killer for executed processes. Takes an integer between -1000 (to disable OOM killing for this process) and 1000 (to make killing of this process under memory pressure very likely). See proc.txt[1] for details.

Answer 6

Looking into the code is always a good idea. Normally I use the search function of kernel.org, but its currently down. Github does a good job too. I found this:

Instead, a new tunable, /proc/pid/oom_score_adj, is added that ranges from -1000 to +1000. It may be used to polarize the heuristic such that certain tasks are never considered for oom kill while others may always be considered. The value is added directly into the badness() score so a value of -500, for example, means to discount 50% of its memory consumption in comparison to other tasks either on the system, bound to the mempolicy, in the cpuset, or sharing the same memory controller.

From https://github.com/mirrors/linux-2.6/commit/a63d83f427fbce97a6cea0db2e64b0eb8435cd10#include/linux/oom.h

Answer 7

The files /proc/<pid>/oom_adj and /proc/<pid>/oom_score_adj adjust the same setting in the kernel and you can modify either one. The difference is that the older interface oom_adj uses more coarse scale from -16 to 15 whereas newer oom_score_adj uses scale from -1000 to 1000.

You can verify that both files control the same setting like this (in shell):

$ cd /proc/$$
$ grep . oom_*
oom_adj:0
oom_score:0
oom_score_adj:0
$ echo 500 > oom_score_adj
$ grep . oom_*
oom_adj:8
oom_score:500
oom_score_adj:500
$ echo 10 > oom_adj
$ grep . oom_*
oom_adj:9
oom_score:588
oom_score_adj:588

I used Linux kernel version 5.4.119 which seems to have off-by-one behavior for the older interface. I'm not sure if this intentional or not. The read-only file oom_score returns the actually used value which should be identical to oom_score_adj.

Setting negative values for either file requires root access so this can be used only to volunteer to be killed in case of system running out of memory (OOM).

The way oom_score works is checking the actual memory usage of a process and then counting 1000 * memory_usage_of_process / total_usable_memory_in_system + oom_score to get a comparision number for every process. The higher the number you get, the more probable its for OOM Killer to select the given process (in addition to memory usage, the amount of child processes are considered, too). The value of -1000 for oom_score is special because it cannot be selected by OOM Killer no matter the computed value of the above computation. In most cases the resulting value would be negative enough to not be selected in any case.

And you can use either of these to volunteer your shell (script or not) as the volunteer for killing in case system runs out of memory. Note that all new processes launched by a process will inherit the same oom_score_adj value by default so you can basically write a shell script that does

#!/bin/bash
echo 1000 > /proc/$$/oom_score_adj
the-command-that-should-be-killed-if-oom
echo 0 > /proc/$$/oom_score_adj

That way only the-command-that-should-be-killed-if-oom (and it's subprocesses) will end up with oom_score_adj value 1000 and your script is still using the value 0 after the third line. You can obviously skip the last line if the script doesn't need to do anything after the process exits – in that case you should also use exec to save some RAM by not keeping the shell script alive after starting the process.