2

Need for some expert advise has brought me here.

Observation: OS locks up during high RAM usage, very low swap usage (plenty available), high tmpfs usage.

My setup:

  1. Ubuntu Server 16.04.5 with kernel 4.4.0-131-generic.
  2. 64GB RAM, 200GB swap partition on a 1TB NVMe SSD.

  3. 64GB tmpfs partition with following fstab entry

    tmpfs  /my-tmp  tmpfs  rw,size=64G,noexec,nosuid,nodev,noatime 0 0

  4. following entries in /etc/sysctl.conf

    vm.overcommit_memory=2
vm.overcommit_ratio=100
vm.swappiness=40

^ I've tried swappiness values of 1, 20, 40 and the default 60 without much success.

The setup is like this to support multiple high performance services, which need high speed temporary filesystem access as well as large amounts of RAM.

Everything works fine until both RAM usage and tmpfs usage are close to maxing out and the OS just locks up (no ssh response, no ability to Ctrl+Alt+F2/3/4, Num Lock does not toggle etc.) for an extended period of time (like 20 minutes), and one of my processes crashes if that happens.

I've logged free memory and tmpfs usage to a file and noticed that when this happens, swap usage is very low, probably because tmpfs usage is not counted against RAM usage, since "used" Mem shows such a small value.

Picture of memory usage looks something like this when the system locks up

              total        used        free      shared  buff/cache   available
Mem:            62G        1.2G        324M         60G         61G         75M
Swap:          186G        4.0G        182G

Filesystem      Size  Used Avail Use% Mounted on
tmpfs           6.3G  9.1M  6.3G   1% /run
tmpfs            32G   20M   32G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            32G     0   32G   0% /sys/fs/cgroup
tmpfs           8.0G  8.0K  8.0G   1% /tmp
tmpfs            64G   61G  3.4G  95% /my-tmp
tmpfs           6.3G  4.0K  6.3G   1% /run/user/1001

Let me preempt some comments about making my services more efficient... they already are. It's just a very very demanding application. And I figured with such a large swap partition, it would be OK.

Troubleshooting performed so far:

  1. Read tmpfs kernel documentation here (many times over)

http://man7.org/linux/man-pages/man5/tmpfs.5.html

It says tmpfs can employ swap space during high memory pressure, but I've seen little to no swap usage when the system locks up, instead of starting to swap.

  1. Read numerous help pages, including this one

https://unix.stackexchange.com/questions/271211/do-tmpfs-and-devtmpfs-share-the-same-memory-region/271235#271235

Hypothesis:

If I could have tmpfs usage count as memory usage, then maybe the vm.swappiness parameter would come into play before such lock-up situation.

Or, if I could independently set swappiness of tmpfs, then that would do the trick as well.

Other suggestions are welcome as well.

Thank you for taking time to read this.

Edit 2018-10-11 09:00 PDT:

Thank you everyone for your helpful responses. I have tried several combinations of swap partition size and tmpfs partition size in the past. Every time, I've observed the same. tmpfs uses up all the RAM, which is not counted as RAM usage (and hence vm.swappiness did not apply to it), and caused OS to lockup. If I make my tmpfs partition smaller than the ram, say 50GB, then lockups don't happen.

This is contrary to kernel documentation for ramfs and tmpfs (which is why I never used ramfs)

https://www.kernel.org/doc/Documentation/filesystems/ramfs-rootfs-initramfs.txt

https://www.kernel.org/doc/Documentation/filesystems/tmpfs.txt

The bottom line is, if the total virtual memory in my system (RAM + SWAP) is larger than total size of all tmpfs partitions in my system, there should not be any OS lockups. That is the expectation, unless I am missing something. I understand things may slow down due to swapping, but should not lock-up.

Edit 2018-10-25 09:15 PDT:

Bump. The original question is not resolved.

Reducing the size of tmpfs partition is not the "solution" I was looking for. I was expecting a way to maximize the size of tmpfs partition to the limits of available virtual memory in my system (RAM + swap - other size of tmpfs partitions) and the system manage swapping of data on tmpfs partition without locking up, as per the kernel documentation.

|improve this question
  • 1
    You cannot swap out a tmpfs. swap just works on memory pages that are paged out to disk. If a process does need pages/memory that is swapped out, the pages/memory are first paged in/copied back to RAM before a process can access it. swapped out stuff is not read directly from swap, it first has to be copied to RAM. – Thomas Oct 11 '18 at 4:51
  • Thank you. It makes sense. The hope was to not swap-out the entire tmpfs, but have it treated just like virtual memory, where same algorithms which decide what to swap out, apply to data in my tmpfs partition as well. Currently, it appears that tmpfs uses all the RAM it can, leaving rest of the system with no memory and hence the OS locks up. This is similar to ramfs, which is why I didn't use it in my setup. This tmpfs behavior is contrary to the kernel documentation which says that tmpfs could employ swap if memory pressure is high. – madhurya Oct 11 '18 at 15:45
  • I just re-read kernel documentation for ramfs and tmpfs. And based on that, my situation should not happen. It's either a bug or I'm missing some other configuration parameter. kernel.org/doc/Documentation/filesystems/tmpfs.txt. kernel.org/doc/Documentation/filesystems/… – madhurya Oct 11 '18 at 16:01
  • Ah, you are right. tmpfs can bei swapped out. Nur since you have allocated 100% of RAM to tmpfs, the system will deadlock. – Thomas Oct 11 '18 at 16:56
0

I'd try...

  • reduce swap to 8G

  • reduce tmpfs to 32G

  • comment out modified vm.* sysctl values:

    # vm.overcommit_memory=2

    # vm.overcommit_ratio=100

    # vm.swappiness=40

  • change your /etc/fstab entry for 32G:

    tmpfs /my-tmp tmpfs rw,size=32G,noexec,nosuid,nodev,noatime 0 0

  • reboot

  • play with the vm.swappiness parameter
  • note any improvement or degradation in system performance

Update #1:

Reducing the tmpfs from 64G to 32G seems to work. You don't want tmpfs to consume all of the physical RAM.

|improve this answer
  • Thank you. Without the the tmpfs my-tmp (commenting out that line in fstab), there are no lockups but my services slow down to a crawl, as expected due to disk IO being way slower than tmpfs. Unfortunately, that's not a favorable option. – madhurya Oct 11 '18 at 1:22
  • It's not favorable because these lockups only happen in extreme situations where slowdown is acceptable due to swapping but in normal situations, the best possible performance is expected. – madhurya Oct 11 '18 at 1:43
  • What happens if you change your swap to 8G and tmpfs to 32G and reboot? – heynnema Oct 11 '18 at 3:14
  • I've tried making tmpfs smaller (with same swap partition). All is well. My understanding is that with a smaller tmpfs size limit, tmpfs does not fill up all the RAM, which remains available for processes to use as virtual memory and hence, the system does not lockup. If we use a smaller tmpfs partition though, it is detrimental to the performance again :(. – madhurya Oct 11 '18 at 15:42
  • @madhurya but making tmpfs the same size as physical RAM has got to cause a problem. I'd still reduce your swap. Keep in mind that unused RAM is turned into a file/disk buffer. I've edited my answer. Please remember to accept it if it was helpful. Thanks. – heynnema Oct 11 '18 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.