21

I have a cloud server with ~14G of RAM and no swap. However, I occasionally see kswapd0 taking up some CPU when I run top. Why would kswapd0 be running at all if there's no swap space for it to manage?

8

It still has a process to check if there's any swap. To reduce it, you'll need to set your swappiness -

edit "/etc/sysctl.conf" as root, then change (or add)

vm.swappiness = 0
  • 3
    Ok, but why is it using 1% of my cpu? – portforwardpodcast Jun 10 '15 at 23:00
  • 2
    if kswapd0 is taking any CPU and you do not have swap, the system is nearly out of RAM and is trying to deal with the situation by (in practise) swapping pages from executables. The correct fix is to reduce workload, add swap or (preferably) install more RAM. Adding swap will improve performance because kernel will have more options about what to swap to disk. Without swap the kernel is practically forced to swap application code. – Mikko Rantalainen Feb 15 '18 at 13:12
  • If you have swap enabled and kswapd0 is using some CPU and you do not want that, lower the swappiness setting. However, unless your swap is backed by SSD that suffers from writing (e.g. bad wear leveling algorithm), lowering the swappiness reduces the system overall performance. The idea is to keep a copy of RAM in the swap in case more RAM is needed - in that case the copy in RAM is thrown away immediately instead of starting to swap that out before the RAM can be used. This optimistic swapping is only done while system is idle enough so it should never slow down your system. – Mikko Rantalainen Feb 15 '18 at 13:15
26

Swap space is only used for data that is not backed by any other file. Data that is mapped from other files on disk ( such as executable programs ) is still swapped to their respective files even if you don't have a swap device.

  • 9
    For example, consider a case where you have zero swap and system is nearly running out of RAM. The kernel will take memory from e.g. Firefox (it can do this because Firefox is running executable code that has been loaded from disk - the code can be loaded from disk again if needed). If Firefox then needs to access that RAM again N seconds later, the CPU generates "hard fault" which forces Linux to free some RAM (e.g. take some RAM from another process), load the missing data from disk and then allow Firefox to continue as usual. This is pretty similar to normal swapping and kswapd0 does it. – Mikko Rantalainen Feb 15 '18 at 13:08
3

It is a well known problem that when Linux run out of memory it can enter swap loops instead of doing what it should be doing, killing processes to free up ram. There are an OOM (Out of Memory) killer that does this but only if Swap and RAM are full.

However this should not really be a problem. If there are a bunch of offending processes, for example Firefox and Chrome, each with tabs that are both using and grabbing memory, then these processes will cause swap read back. Linux then enters a loop where the same memory are being moved back and forth between memory and hard drive. This in turn cause priority inversion where swapping a few processes back and forth makes the system unresponsive.

If you disable swap you make this problem worse as kswapd0 now have no option than to swap out mapped memory such as executables. If you swap out executables it is even more likely that they will be swapped back in again rather quickly.

I tried triggering this behavior in NetBSD for testing and what happened there is that the the offending process became incredible slow while the OS itself was very responsive. Meaning that the swapping problem do occur but there are no priority inversion. However NetBSD doesn't have AMDGPU drivers so I am sticking to Linux for time being. Perhaps NetBSD doesn't memory map executables and that is why it doesn't enter swap loops but I don't really know enough about it's implementation to say why it doesn't become unresponsive.

Facebook had this problem as well and created the OOMD which is the Out Of Memory Daemon. This is daemon that detects kswapd0 activity and starts killing processes. And according to Facebook this almost entirely removed the problem of Linux servers becoming unresponsive. However I have not tested it and I don't know how well it will work on other servers or desktop/laptops. Appealingly OOMD has some logic deciding what processes to kill first in order to preserve system processes and the part of their server system that are responsible for relaunching whatever was killed.

However this is not how it should be solved. OOMD is an UGLY HACK. The real solution is to fix the priority inversion that a swap loop causes as well as making the kernel OOM Killer more aggressive at killing processes to free memory. The fix belong in the kernel because that's the only place where it we can be sure that the problem is detected in time and processes are being properly killed.

Setting swappiness=0 is no solution because when the system is out of free RAM it starts swapping no matter what. There are no option to guarantee that the system doesn't start swapping.

And also fixing the offending applications is not a fix. Expecially not if a user wants to exploit this bug in order to intentionally make the OS unresponsive. To be responsive is the responsibility of the kernel. If Firefox makes itself unresponsive then the fix is to the application. However it is not only making itself unresponsive but causing the entire OS to become very slow and unresponsive. To the level that it can take half an hour to log in to SSH. The SSH has nothing to do with and if it doesn't get to run that is a bug in the kernel, not in any other part of the system. And it is not a bug it is two bugs. One bug is priority inversion where a off the rails swapping cycle is allowed to interfere with other processes than the offending process(es) and that in itself is bad. The other bug is that it doesn't detect that it is in a swap loop and that causes insane wear on the HDD/SSD or whatever storage that is backing the swap. When swapping executable this is less of a problem as they are read only memory maps which aren't written back to disks but kswapd0 still gets locked reading back what it at the same time are deleting from memory.

Oh and there is a third bug. The fact that there are no way to protect disk CACHE from being eaten when memory hungry applications swallow all available memory. This is one of the reasons that kswapd0 makes the system unresponsive. The most hot memory mapped data are usually stored in the disk cache but when firefox has eaten that cache, well it obviously means that disk reads will have to occur.

It's not necessarily Firefox that are causing your problem but it is the default browser, not Chrome. And both are widly known to trigger this problem as they treat available memory as something that is wasted, including cache and swap memory which in Linux counts as "available memory". So in order not to get "available memory" get's wasted it use it for caching and other stuff. Obviously using SWAP for DISK CACHE is a VERY BAD IDEA but the fellows at both Firefox and Chrome respond to that with "free memory is wasted memory".

So what we have here is three kernel bugs that the kernel team do not seem to consider bugs. And a bug in Firefox, Chrome and all derivatives that they do not consider a bug. I tried building Firefox on my Fedora laptop in order to look into this problem and perhaps patch it. Guess what. Building Firefox with GCC on a 4 core CPU with 4GB ram triggers a SWAP LOOP with PRIORITY INVERSION. So one of the applications that has to be rewritten is GCC. On NetBSD what happens is just the 4 running instances of GCC gets slower than running one instance but it doesn't freeze the system.

Yeah this is a bit of a rant but I hope that it clarifies the current problem with the Linux memory subsystems as well as the applications that cause it.

1

If you have no swap and kswapd0 is running, your system is actually using nearly all of the RAM at that moment. It's time to get better tools to monitor memory usage (or free/available memory in the system).

The real fix is to reduce memory usage (run processes with less memory leaks, run less processes, skip running some processes at all, limit number of children/worker processes of some server software) or to get more RAM. If the need for RAM is caused by memory leaks, you may opt to use swap instead. Linux should be pretty smart getting the leaked parts to swap given enough time. Having swap is better than nothing but that is not a real substitute for having adequate amount of RAM.

  • There is good information here as well as in your comments, but enabling swap isn't a solution in the limit where all of the available memory (ram + swap) is getting filled. It's a particularly bad solution in the case of a memory leak, because it's inevitable that all memory will eventually become full. The result when swap+ram is full is the same as when ram is full and swap is disabled. – Code Bling Nov 4 at 2:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.