Need for some expert advise has brought me here.
Observation: OS locks up during high RAM usage, very low swap usage (plenty available), high tmpfs usage.
My setup:
- Ubuntu Server 16.04.5 with kernel 4.4.0-131-generic.
- 64GB RAM, 200GB swap partition on a 1TB NVMe SSD.
64GB tmpfs partition with following fstab entry
tmpfs /my-tmp tmpfs rw,size=64G,noexec,nosuid,nodev,noatime 0 0
following entries in /etc/sysctl.conf
vm.overcommit_memory=2 vm.overcommit_ratio=100 vm.swappiness=40
^ I've tried swappiness values of 1, 20, 40 and the default 60 without much success.
The setup is like this to support multiple high performance services, which need high speed temporary filesystem access as well as large amounts of RAM.
Everything works fine until both RAM usage and tmpfs usage are close to maxing out and the OS just locks up (no ssh response, no ability to Ctrl+Alt+F2/3/4, Num Lock does not toggle etc.) for an extended period of time (like 20 minutes), and one of my processes crashes if that happens.
I've logged free memory and tmpfs usage to a file and noticed that when this happens, swap usage is very low, probably because tmpfs usage is not counted against RAM usage, since "used" Mem shows such a small value.
Picture of memory usage looks something like this when the system locks up
total used free shared buff/cache available
Mem: 62G 1.2G 324M 60G 61G 75M
Swap: 186G 4.0G 182G
Filesystem Size Used Avail Use% Mounted on
tmpfs 6.3G 9.1M 6.3G 1% /run
tmpfs 32G 20M 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
tmpfs 8.0G 8.0K 8.0G 1% /tmp
tmpfs 64G 61G 3.4G 95% /my-tmp
tmpfs 6.3G 4.0K 6.3G 1% /run/user/1001
Let me preempt some comments about making my services more efficient... they already are. It's just a very very demanding application. And I figured with such a large swap partition, it would be OK.
Troubleshooting performed so far:
- Read tmpfs kernel documentation here (many times over)
http://man7.org/linux/man-pages/man5/tmpfs.5.html
It says tmpfs can employ swap space during high memory pressure, but I've seen little to no swap usage when the system locks up, instead of starting to swap.
- Read numerous help pages, including this one
Hypothesis:
If I could have tmpfs usage count as memory usage, then maybe the vm.swappiness parameter would come into play before such lock-up situation.
Or, if I could independently set swappiness of tmpfs, then that would do the trick as well.
Other suggestions are welcome as well.
Thank you for taking time to read this.
Edit 2018-10-11 09:00 PDT:
Thank you everyone for your helpful responses. I have tried several combinations of swap partition size and tmpfs partition size in the past. Every time, I've observed the same. tmpfs uses up all the RAM, which is not counted as RAM usage (and hence vm.swappiness did not apply to it), and caused OS to lockup. If I make my tmpfs partition smaller than the ram, say 50GB, then lockups don't happen.
This is contrary to kernel documentation for ramfs and tmpfs (which is why I never used ramfs)
https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt
https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt
The bottom line is, if the total virtual memory in my system (RAM + SWAP) is larger than total size of all tmpfs partitions in my system, there should not be any OS lockups. That is the expectation, unless I am missing something. I understand things may slow down due to swapping, but should not lock-up.
Edit 2018-10-25 09:15 PDT:
Bump. The original question is not resolved.
Reducing the size of tmpfs partition is not the "solution" I was looking for. I was expecting a way to maximize the size of tmpfs partition to the limits of available virtual memory in my system (RAM + swap - other size of tmpfs partitions) and the system manage swapping of data on tmpfs partition without locking up, as per the kernel documentation.
tmpfs
. swap just works on memory pages that are paged out to disk. If a process does need pages/memory that is swapped out, the pages/memory are first paged in/copied back to RAM before a process can access it. swapped out stuff is not read directly from swap, it first has to be copied to RAM.