5

My Ubuntu 18.04 (Bionic Beaver) installation crashed because of a problem with out of memory.

I have messages like this:

Jan 31 16:56:53 adam-Lenovo-YOGA-720-15IKB kernel: [47980.843963] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=chrome,pid=16144,uid=1000
Jan 31 16:56:53 adam-Lenovo-YOGA-720-15IKB kernel: [47980.843992] Out of memory: Killed process 16144 (chrome) total-vm:1229048kB, anon-rss:84920kB, file-rss:0kB, shmem-rss:140kB
Jan 31 16:56:53 adam-Lenovo-YOGA-720-15IKB kernel: [47980.895646] oom_reaper: reaped process 16144 (chrome), now anon-rss:0kB, file-rss:0kB, shmem-rss:140kB
Jan 31 17:07:20 adam-Lenovo-YOGA-720-15IKB kernel: [    0.000000] microcode: microcode updated early to revision 0xca, date = 2019-10-03
Jan 31 17:07:20 adam-Lenovo-YOGA-720-15IKB kernel: [    0.000000] Linux version 5.3.0-28-generic (buildd@lcy01-amd64-009) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #30~18.04.1-Ubuntu SMP Fri Jan 17 06:14:09 UTC 2020 (Ubuntu 5.3.0-28.30~18.04.1-generic 5.3.13)
Jan 31 17:07:20 adam-Lenovo-YOGA-720-15IKB kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.3.0-28-generic root=UUID=e8808971-420e-45c4-96d7-2f490862bf41 ro quiet splash vt.handoff=1
Jan 31 17:07:20 adam-Lenovo-YOGA-720-15IKB kernel: [    0.000000] KERNEL supported cpus:

I have all the times a lot of programs running (Firefox with multiple windows and tabs, Chromium, Chrome, GIMP, Emacs, LibreOffice, and VMware Player for Windows 7).

Today I have 8 GB of RAM and 8 GB of swap file:

adam@adam-Lenovo-YOGA-720-15IKB:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:           7.5G        3.7G        351M        692M        3.4G        2.8G
Swap:          8.0G          0B        8.0G

Can we have some dynamic swap on Linux instead of limiting it to a preallocated size?

Swappiness:

adam@adam-Lenovo-YOGA-720-15IKB:~$ sysctl vm.swappiness
vm.swappiness = 10
  • Edit your question and show me sysctl vm.swappiness. – heynnema Jan 31 at 17:02
  • @heynnema I have seen a video on YT and changed it actually: youtu.be/BLVtxpm5c2A?t=660 – London Smith Jan 31 at 17:17
  • 2
    the suggestion is to enable zram. It's much faster than normal disk swap. See also Does Ubuntu support dynamic swap file sizing?. And it's GB or GiB, not Gb which means Gigabit and is only 1/8 of the GB – phuclv Feb 1 at 4:00
  • I've made a minor correction in my answer. Please note. – heynnema Feb 1 at 13:42
  • When you hit an OOM-killer condition, was most of your swap space full? Linux shouldn't OOM kill anything until it runs out of swap space, for any non-zero swappiness. (Did you accept heynnema's answer just for (correctly) saying you need more RAM to run your workload, or did the swappiness change really make a difference?) And BTW, the log you showed didn't "crash Ubuntu", it crashed one big program running under Ubuntu. Do you sometimes find your X server or window manage getting OOM killed? – Peter Cordes Feb 1 at 15:56
7

You watched a YouTube video that discussed the parameter vm.swappiness, and then set your vm.swappiness=10, which is totally wrong for your system with 8G RAM. That's why you're "running out of memory".

Set vm.swappiness=80 (based on 8G RAM and 8G SWAP), this way...

sudo -H gedit /etc/sysctl.conf # edit this file

Search for an existing vm.swappiness= entry...

CTRL+f vm.swappiness

  • If found, edit it to say vm.swappiness=80

  • If not found, add vm.swappiness=80 at the end of the file

Save your edits and quit gedit

sudo sysctl -p

Note: Based on what apps you're running, you might also be best served by adding more RAM, and changing vm.swappiness to a different value. To get the speed advantage of memory interleaving, do it in equal size RAM stick pairs, so in your case, add another 8G RAM stick (if you have one 8G RAM stick installed now). If you have two 4G sticks installed now, then you'll need two 8G RAM sticks.

| improve this answer | |
  • Your second command would overwrite what the first one did if there's a setting for vm.swappiness somewhere in /etc/sysctl.*. – Ruslan Feb 1 at 7:56
  • @Ruslan The first command writes the value to /etc/sysctl.conf, the second command "Load in sysctl settings from the file specified or /etc/sysctl.conf if none given", as per man sysctl. – heynnema Feb 1 at 12:40
  • No, it never changes the files. What this option (effectively) writes to is /proc/sys/vm/swappiness, which doesn't preserve its values across reboot — only applies to the currently-running kernel. – Ruslan Feb 1 at 12:42
  • 1
    Do you have any evidence that Linux will invoke the OOM killer instead of swapping if there is swap available, for any non-zero swappiness? That hasn't been my experience; low swappiness just favours keeping anonymous pages of processes over more space for the pagecache. Agreed that 8GB is simply not enough RAM (and more than 8GB of swap might avoid OOM, but not with acceptable performance). But I highly doubt that changing swappiness without changing anything else would avoid the OOM killer, unless there have been massive changes to what it means in the last couple years. – Peter Cordes Feb 1 at 15:53
  • 1
    I'm with @PeterCordes, this answer is wrong except for the last paragraph. Non-zero swappiness will never result in the OOM killer invoked, it merely controls the eagerness of choosing swap versus RAM. From the OOM point of view, RAM and swap are the same thing, a machine can't "run out of memory" while there's some swap space available. – Sergey Feb 5 at 0:39
5

An answer from Out of swap - what happens? explains dynamically sizing swap space is perilous and offers a way to test that hypothesis.

There was a package, SwapSpace, which you may run into, but it is no longer maintained.

An alternative dphys-swapfile service, detailed at Dynamically growing swap file on Debian may not be as dynamic as you need.

The solution from heynnema is superior and should be implemented before anything else.

| improve this answer | |
  • If @RonJohn's answer works (creating a swap file with fallocate instead of dd), dynamic swap file creation barely adds any I/O pressure when the system is running low on memory. The answer you linked says that swapping at all is perilous and recommends not using any swap space. That's not an option for the OP unless they add more RAM; they're hitting the OOM killer with 8GB of always-available swap. – Peter Cordes Feb 1 at 15:48
3

Naturally, adding more RAM is the best solution.

However, sometimes that's impractical, as it used to be for me. Because of that, I would sometimes enable a swap file when the system started to get really sluggish. That would give me some time to kill what I wanted killed.

https://linuxize.com/post/create-a-linux-swap-file/

  1. sudo fallocate -l 4G /swapfile # or whatever you want to name it, and wherever you want to place it.
  2. sudo chmod 600 /swapfile
  3. sudo mkswap /swapfile

Then, when the system gets sluggish, run sudo swapon /swapfile. When you deem it not necessary anymore, run sudo swapoff /swapfile.

EDIT: fallocate is know to work on ext4, but fail on XFS. dd if=/dev/zero of=/swapfile bs=1M count=4096 works on XFS.

| improve this answer | |
  • When did mkswap / swapon start working on preallocated unwritten extents? It didn't used to; that's a really nice change that makes on-the-fly swap file creation practical. (e.g. auto-managed by a daemon; Dynamically growing swap file on Debian) – Peter Cordes Feb 1 at 15:45
  • @PeterCordes does my answer imply that it does? I don't think so, since Step #1 says to preallocate a swap file, but maybe wasn't clear. – RonJohn Feb 1 at 16:53
  • Yeah, fallocate -l preallocates but doesn't zero the underlying extents. These are what's called "unwritten extents"; they read as zero when read through the FS (like holes), but the kernel doesn't do that after swapon. That's why this didn't used to work. (But of course exposing previous non-zeroed free space contents is not actually a security risk; we're exposing it to the kernel.) Oh, apparently this has worked for a while with EXT4. It was a problem with XFS, maybe still is, see this comment for links to a mailing list post. – Peter Cordes Feb 1 at 18:59
  • @PeterCordes interesting. I just confirmed that while it does work with ext4, sudo swapon with an XFS volume results in swapon: /Data/swapfile: swapon failed: Invalid argument. Thanks. – RonJohn Feb 1 at 19:35
  • After you dd or fallocate your file, chmod it, and mkswap it.. if you want Linux to keep using it after reboot, for a swapfile called /swapfile you can add a line to /etc/fstab like "/swapfile none swap sw 0 0" . It's then turned on automatically during boot. Sometimes I've preferred leaving a swapfile out of fstab, I just put "swapon /swapfile" into /etc/rc.local instead. – user153822 Feb 28 at 2:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.