Ubuntu 16.04 never swapping on

Question

I am using Xubuntu 16.04 on my laptop, installed from a dist-upgrade of a Xubuntu 14.

The laptop has 10 GB RAM and 4 GB Swap.

Recently I upgraded the kernel to version: 4.15.0-51.

The problem is that during normal operation the system is swapping-out over time but never swapping back in. So after hours/days of uptime, the swap space gets full.

Even after closing all the main processes the RAM usage goes to a few hundred MB but the swap remains full. Also, looping over all the /proc/*/stats and grep VmSwap return 0 kB for all processes.

How can I debug why the kernel is not swapping in?

It's like the kernel is losing track of what processes the pages on the swap belong to.

I've already invested a significant amount of time trying to fix this with no success. The only turn around I have for now is a script that is creating and recycling (swapoff ; swapon) a few swap files.

Thank you for any help.

Answer 1

Today I gave it another try and found the problem.

Started by checking again the meminfo information. Here is the output of cat /proc/meminfo:

MemTotal:       10133276 kB
MemFree:         2617372 kB
MemAvailable:    4660420 kB
Buffers:          503472 kB
Cached:          3825884 kB
SwapCached:        59448 kB
Active:          3852624 kB
Inactive:        3112996 kB
Active(anon):    2688920 kB
Inactive(anon):  2245220 kB
Active(file):    1163704 kB
Inactive(file):   867776 kB
Unevictable:          32 kB
Mlocked:              32 kB
SwapTotal:       6096888 kB
SwapFree:        2644324 kB
Dirty:                52 kB
Writeback:             0 kB
AnonPages:       2576972 kB
Mapped:           770460 kB
Shmem:           2297836 kB
Slab:             391072 kB
SReclaimable:     325960 kB
SUnreclaim:        65112 kB
KernelStack:       15824 kB
PageTables:        53492 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    11163524 kB
Committed_AS:   12259512 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:      2048 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      349640 kB
DirectMap2M:    10039296 kB

Despite the issue of having 4 GB of swap used by nothing, this time the 2GB of shmem used caught my attention.

Searching for that, I found this question with the answer: High SHMem memory usage!

So, in short, having high shmem usage can be related to tmpfs directories. In my case, /dev was using almost 4.8 GB

Here is the output of df -h:

Filesystem      Size  Used Avail Use% Mounted on
udev            4,8G  4,8G     0 100% /dev
tmpfs           990M   34M  956M   4% /run
/dev/sda2       255G  232G   11G  96% /
tmpfs           4,9G  220M  4,7G   5% /dev/shm
tmpfs           5,0M  4,0K  5,0M   1% /run/lock
tmpfs           4,9G     0  4,9G   0% /sys/fs/cgroup
/dev/sda5       333G  208G  126G  63% /media/dados
cgmfs           100K     0  100K   0% /run/cgmanager/fs
tmpfs           990M   88K  990M   1% /run/user/1000

After that I just searched in /dev for the culprit using the du command. As also referred to in the thread I linked above, it was the bootchart command that was storing huge log files inside the /dev directory.

Thanks for the help.

Answer 2

You don't need to worry about it. Have a look for example here. Kernel is tracking it and it can reause space in swap.