vm.swappiness does not apply to tmpfs, Ubuntu Server 16.04.5

Question

Need for some expert advise has brought me here.

Observation: OS locks up during high RAM usage, very low swap usage (plenty available), high tmpfs usage.

My setup:

1. Ubuntu Server 16.04.5 with kernel 4.4.0-131-generic.
2. 64GB RAM, 200GB swap partition on a 1TB NVMe SSD.

3. 64GB tmpfs partition with following fstab entry

   tmpfs  /my-tmp  tmpfs  rw,size=64G,noexec,nosuid,nodev,noatime 0 0
   

4. following entries in /etc/sysctl.conf

   vm.overcommit_memory=2
   vm.overcommit_ratio=100
   vm.swappiness=40
   

^ I've tried swappiness values of 1, 20, 40 and the default 60 without much success.

The setup is like this to support multiple high performance services, which need high speed temporary filesystem access as well as large amounts of RAM.

Everything works fine until both RAM usage and tmpfs usage are close to maxing out and the OS just locks up (no ssh response, no ability to Ctrl+Alt+F2/3/4, Num Lock does not toggle etc.) for an extended period of time (like 20 minutes), and one of my processes crashes if that happens.

I've logged free memory and tmpfs usage to a file and noticed that when this happens, swap usage is very low, probably because tmpfs usage is not counted against RAM usage, since "used" Mem shows such a small value.

Picture of memory usage looks something like this when the system locks up

              total        used        free      shared  buff/cache   available
Mem:            62G        1.2G        324M         60G         61G         75M
Swap:          186G        4.0G        182G

Filesystem      Size  Used Avail Use% Mounted on
tmpfs           6.3G  9.1M  6.3G   1% /run
tmpfs            32G   20M   32G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            32G     0   32G   0% /sys/fs/cgroup
tmpfs           8.0G  8.0K  8.0G   1% /tmp
tmpfs            64G   61G  3.4G  95% /my-tmp
tmpfs           6.3G  4.0K  6.3G   1% /run/user/1001

Let me preempt some comments about making my services more efficient... they already are. It's just a very very demanding application. And I figured with such a large swap partition, it would be OK.

Troubleshooting performed so far:

1. Read tmpfs kernel documentation here (many times over)

http://man7.org/linux/man-pages/man5/tmpfs.5.html

It says tmpfs can employ swap space during high memory pressure, but I've seen little to no swap usage when the system locks up, instead of starting to swap.

2. Read numerous help pages, including this one

https://unix.stackexchange.com/questions/271211/do-tmpfs-and-devtmpfs-share-the-same-memory-region/271235#271235

Hypothesis:

If I could have tmpfs usage count as memory usage, then maybe the vm.swappiness parameter would come into play before such lock-up situation.

Or, if I could independently set swappiness of tmpfs, then that would do the trick as well.

Other suggestions are welcome as well.

Thank you for taking time to read this.

Edit 2018-10-11 09:00 PDT:

Thank you everyone for your helpful responses. I have tried several combinations of swap partition size and tmpfs partition size in the past. Every time, I've observed the same. tmpfs uses up all the RAM, which is not counted as RAM usage (and hence vm.swappiness did not apply to it), and caused OS to lockup. If I make my tmpfs partition smaller than the ram, say 50GB, then lockups don't happen.

This is contrary to kernel documentation for ramfs and tmpfs (which is why I never used ramfs)

https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt

https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt

The bottom line is, if the total virtual memory in my system (RAM + SWAP) is larger than total size of all tmpfs partitions in my system, there should not be any OS lockups. That is the expectation, unless I am missing something. I understand things may slow down due to swapping, but should not lock-up.

Edit 2018-10-25 09:15 PDT:

Bump. The original question is not resolved.

Reducing the size of tmpfs partition is not the "solution" I was looking for. I was expecting a way to maximize the size of tmpfs partition to the limits of available virtual memory in my system (RAM + swap - other size of tmpfs partitions) and the system manage swapping of data on tmpfs partition without locking up, as per the kernel documentation.

Answer 1

I'd try...

• reduce swap to 8G

• reduce tmpfs to 32G

• comment out modified vm.* sysctl values:

  # vm.overcommit_memory=2

  # vm.overcommit_ratio=100

  # vm.swappiness=40

• change your /etc/fstab entry for 32G:

  tmpfs /my-tmp tmpfs rw,size=32G,noexec,nosuid,nodev,noatime 0 0

• reboot

• play with the vm.swappiness parameter
• note any improvement or degradation in system performance

Update #1:

Reducing the tmpfs from 64G to 32G seems to work. You don't want tmpfs to consume all of the physical RAM.