82

kswapd0 is taking 99.9% of my CPU as top shows me, the problem appeared today when gaming and first time it went away after 6 minutes and now it has been doing it for about 20 minutes. How is this fixable and what is causing this?

6
  • 1
    This is happening to me for 18.04 as well. Details here: askubuntu.com/questions/1118932/… Feb 17 '19 at 10:23
  • 1
    This happens primarily when I try to do something in the Steam client after I wake up my computer. I have 32 Gigs on my machine on my Kubuntu install, with the swap turned off. It makes no sense that kswapd0 would go on overload, so much so the system is almost completely frozen!!!!
    – flajann
    May 22 '20 at 18:36
  • If you're not using swap, disable it with swapoff -a. Run it as a superuser. Dec 16 '20 at 7:57
  • 2
    I recently ran into this issue and this can also be due to a crypto miner. More info here: yoroi.company/research/…. Hope this helps someone! Apr 6 at 9:20
  • @flajann It sounds like you are running out of RAM. Disabling swap is almost never the solution unless it's absolutely necessary as swappiness prevents a frozen system unless you run out of swap space.
    – mchid
    Jul 19 at 16:31
83

The process kswapd0 is the process that manages virtual memory. Your machine should have RAM, SWAP, and the EXT4 on your HDD/SSD. The ext4 is where everything is stored, and it is always slower to access than RAM. RAM is like a half-way running space for programs to access information quickly. Most computers have at least 4GB of RAM, which under normal conditions is plenty. When playing a game, however, you may run low on RAM space, which is where SWAP comes in.

SWAP is a fake RAM located on your HDD/SSD next to your EXT4. It is quicker to access than the EXT4, but it is much slower than actual RAM. When you run low on memory, kswapd0 moves programs that you are not using/not using as much as other programs to the SWAP, which causes extreme lag on those processes. If your game was needing 5GB RAM, 1GB at LEAST would be in SWAP. That means when it tries to access that information, it has to wait longer to get it.

This entire process causes extreme CPU usage, moving information from and to SWAP and RAM and handling the request of information all at the same time. How to solve this issue?

  1. Tell kswapd0 to only move stuff to SWAP when you are completely OUT of RAM. This is the single most effective method to resolving SWAP issues. Run

    echo vm.swappiness=0 | sudo tee -a /etc/sysctl.conf

    where 0 is the percent left out of 100 at which SWAP should be used (when you have 0% RAM left, SWAP will start taking in data). You can also just edit /etc/sysctl.conf to your liking instead of adding this command to the end of it everytime using gedit or nano or whatever, be sure to sudo though, this file is root owned. Reboot and your are set!

  2. Reduce the consumption of RAM by other processes or close other programs while running high memory programs. This is why most games tell you to close all other windows before playing, or installations do the same. Things like file syncing services tend to take a lot of memory.
  3. Buy more RAM. Installing RAM is not as hard as it sounds. One or two screws on a small compartment (if you are on a laptop) and a simple click. Just be sure you are buying the correct kind!
  4. Lower processes of the CPU much as you did with the RAM. This will help those RAM to SWAP bursts to go much smoother.

That's the best that you can do. Others may say disable swap completely, but that is dangerous and I would NOT recommend that. That can cause entire systems to freeze up if there is a memory leak or too many applications running. Just realize that the SWAP is a failsafe for the RAM. It is definitely not as fast or efficient as RAM, but it's better than Window's Pagefile! (which accomplishes the same purpose)

EDIT: If you are interested in learning more about SWAP, see here.

7
  • I cant exactly remember what fixed the problem for me anymore, but I thank you for the well written answer that explain a lot.
    – Kaspar
    Sep 30 '14 at 19:11
  • According to your answer, I terminate some process so that less swap is used. Now the process kwapd0 is gone. Thanks.
    – mtoloo
    Oct 22 '17 at 13:22
  • Consider your use case when using swap. If running Kubernetes, for example, official docs demand swap be disabled for proper functionality of the kublet service used for system management. Dec 16 '20 at 7:41
  • 1
    @AndrewKoster Interesting, I'd be sure to reboot your system and ensure the swappiness is still correct. If you want to guarantee swap is never used, I would turn off your swap partitions/files. Since your problem might be different that what is asked here (and considering this question is originally over 8 years old), you should open your own question with some memory stats of the problem you're seeing.
    – Zzzach...
    Jun 20 at 6:17
  • 1
    @AndrewKoster that's because the swappiness doesn't disable swap, it's determines how aggressive the kernel should be at preemptively swapping (swapping to virtual memory before running out of RAM). Setting it to zero means that it will not attempt swapping until RAM has run out. Some processes will always allocate virtual memory anyhow (Java processes in particular). The best way to disable swap is with swapoff -a. Aug 20 at 17:39
42

kswapd0 runs at 99.9% of one CPU but is actually not swapping at all

To me it happens at times on Ubuntu 14.04 with kernel 3.19.0-50-generic (and earlier) running in a VMware vm. I have no clue, what made it appear, but it comes during idle time.

top shows:

# top
top - 09:49:35 up 5 days, 18:35,  1 user,  load average: 1.00, 1.00, 0.99
Tasks: 219 total,   2 running, 217 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us, 25.0 sy,  0.0 ni, 74.7 id,  0.2 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem:   3028784 total,  1874468 used,  1154316 free,  1010276 buffers
KiB Swap: 15624188 total,     3032 used, 15621156 free.   234928 cached Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    52 root      20   0       0      0      0 R  99.7  0.0 122:15.21 kswapd0
     3 root      20   0       0      0      0 S   0.3  0.0   0:29.86 ksoftirqd/0
     7 root      20   0       0      0      0 S   0.3  0.0   9:49.47 rcu_sched

Temporary solution

a reboot solved the problem - temporarily.

following the answer on serverfault (kswapd often uses 100% CPU when swap is in use) there where the same settings on my system:

# cat /proc/sys/vm/swappiness
60
# cat /proc/sys/vm/vfs_cache_pressure
100
# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never

the solution was actually running echo 1 > /proc/sys/vm/drop_caches as root user:

# cat /proc/sys/vm/drop_caches
0
# echo 1 > /proc/sys/vm/drop_caches
# cat /proc/sys/vm/drop_caches
1

Or, as pointed out by theTuxRacer (thanks!), use the following command in case you are not logged in as root:

echo 1 | sudo tee /proc/sys/vm/drop_caches

now it's fine:

# top
top - 10:08:58 up 5 days, 18:55,  1 user,  load average: 0.72, 0.95, 0.98
Tasks: 220 total,   1 running, 219 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  0.2 sy,  0.0 ni, 99.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   3028784 total,   681704 used,  2347080 free,     2916 buffers
KiB Swap: 15624188 total,     3032 used, 15621156 free.    81924 cached Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
     9 root      20   0       0      0      0 S   0.3  0.0  14:10.40 rcuos/0
     1 root      20   0   45652   8124   2888 S   0.0  0.3   1:54.98 init

Permanent solution (to be found)?

but since the actual reason is not yet known, and I did not fine any suitable explanation on the net, this is not a permanent solution. Actually, the selected answer could be the permanent solution. I just wanted to add this for future reference, as a reboot (to make sysctl take effect) is not always possible.

An other solution might be to set THP to either madvice or never (see poige's comment to his answer, How do I modify “/sys/kernel/mm/transparent_hugepage/enabled” and the referenced MongoDB Manual on Disable Transparent Huge Pages (THP))

cron job

i've set up the following batch as a cron job as a "permanent" solution:

#!/bin/bash
# Rev 2: Use ps instead of top

## run as cron, thus no $PATH, thus need to define all absolute paths
cpu=$(/usr/bin/printf %.0f $(/bin/ps -o pcpu= -C kswapd0))

[[ -n $cpu ]] \
&& (( $cpu >= 90 )) \
&& echo 1 > /proc/sys/vm/drop_caches \
&& echo "$$ $0: cache dropped (kswapd0 %CPU=$cpu)" >&2 \
&& exit 1

exit 0

invoked from root@localhost:~# crontab -e with

# m h  dom mon dow   command
  * *  *   *   *     /bin/bash /path/to/batch/drop_caches.sh >> /var/log/syslog 2>&1

Note: Above cron job script has been adapted to include Fredrik Erlandsson's suggestion: A much simpler and more efficient way to determine kswapd0's CPU usage. Thanks!

10
  • Very nice answer, thanks. RPi kernel updated and this is what I get, berserking kswap.
    – Paul B
    Jun 1 '16 at 19:27
  • 2
    "echo 1 > /proc/sys/vm/drop_caches" fixed the high CPU usage for me--night and day difference! kswapd0 went from 100% CPU to 0%. An explanation for why and a permanent solution would be great. (Side note: I'm running linux kernel 4.8.0-36-generic with 16 GB mem and 16 GB swap.) Jun 12 '17 at 17:14
  • 2
    I would suggest changing (( $cpu >= 90 )) \ to (( $(printf %.0f `/bin/ps -o pcpu= -C kswapd0`) >= 90 )) \ and removing everything above.. Jan 8 '20 at 21:59
  • 2
    @PeterKrauss, nowadays running Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-74-generic x86_64) and 20.04 LTS (GNU/Linux 5.4.0-21-generic x86_64) I have not experienced this anymore and am not running the cron job mentioned in this answer. May 19 '20 at 9:23
  • 1
    echo 1 | sudo tee /proc/sys/vm/drop_caches in case you are not logged in as root and you have a sluggis terminal. Jun 25 '20 at 15:27
4

A more permanent solution is to increase the size of your swap file.

kswapd will take 100% CPU when searching for swap space that doesn't exist (if your swap partition is full). htop will show if your swap space is full at the top of the screen or you can use the free -h command to check used swap.

After using the following commands to increase the size of my swap file from 4GB to 8GB, kswapd now runs at 0% CPU with swappiness set to 60*.

sudo swapoff /swapfile
sudo rm  /swapfile
sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Click here for a more detailed description of these commands

This assumes that you are already using a swap file (default Ubuntu setup) and that the /swapfile is referenced in /etc/fstab. Also, your swap file may be larger or smaller depending on how much RAM you have so adjust accordingly.

* It should be noted that I use a SSD so disk performance is not an issue. However, swap is almost always a good thing if you're running out of RAM.

2
  • 1
    This is the perfect solution for the issue. Thanks for sharing. Aug 4 at 5:12
  • 2
    "kswapd will take 100% CPU when searching for swap space that doesn't exist" : indeed, for instance when your swap partition is not mounted ! ^^ Oct 2 at 14:56

Not the answer you're looking for? Browse other questions tagged or ask your own question.