0

I am running Ubuntu 14.04LTS on a Dell PowerEdge R730 with 32 GB of RAM and an Nvidia K80 GPU.. I have a python script that runs for a long time, but really just makes a bunch of relatively short-lived system calls to other programs (some of which use CUDA extensively in case that matters).. Other than the overarching python script, the individual system calls only run for about 45 seconds a piece, and the python script itself doesn't "maintain" anything - it just does the system calls and iterates - not any storage of results, etc..

As I watch the program run and monitor memory usage with "top", I see the reported "free" memory go down over time.. I understand this isn't usually a concern due to caching being employed. Eventually, though, the machine hangs and doesn't respond in any way (i.e. no mouse cursor movement on console, no response from remote terminals, etc).. When this happens, I have to hard reboot the system, and things are back to normal, and I can pick up where I left off, until it hangs again..

Another point of interest - after enough reboot-and-restarts the program finally completes.. Again, after this, there is nothing of significance running on the system, however, the gnome-system-monitor and top both report almost full memory usage, and trying to run other (minimal) commands are often "Killed" by the kernel.. Checking the system logs, it reports "out of memory" which seems to indicate that the memory being used is not cache but memory that is actually "claimed". The question is - by who? Checking all the memory utilities that people have mentioned on this and other forums, no processes claim to be using much memory..

I've read reports of kernel memory leaks, and that's what I want to claim, but it just seems unlikely..

Question: How can I determine what is claiming all the memory? I'd like to be able to determine if its in the kernel or if its some process that running.

Supporting Infomation:

uname -a:

Linux machinename 3.19.0-47-generic #53~14.04.1-Ubuntu SMP Mon Jan 18 16:09:14 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

lscpu:

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Stepping:              2
CPU MHz:               1200.351
BogoMIPS:              5993.09
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              10240K
NUMA node0 CPU(s):     0-7

Prior Reading and understanding: I've read through a ton of posts about understanding Linux's memory usage and reporting.. I understand that when programs like top report very little memory "free" it is not necessarily a problem because much of the "used" memory is cached, and actually having your RAM full of cached stuff is good.. I believe, however, that this is not the issue that I am seeing, because if it were cache it seems like programs would be able to make use of it. The fact that the kernel is stepping in and killing new processes and dmesg is reporting the system is "out of memory" seems to indicate memory is being occupied in a non-cache way, but it doesn't seem to be being reported in any memory analysis tool I have tried..

Update: Based on an answer below, I looked at /proc/meminfo when things were starting to go poorly, and while I don't know what all these mean, there are a few that seem .. suspect.. "DirectMap2M" seems way problematic and "VmallocChunk" too, although less so...

> cat /proc/meminfo
MemTotal:       32828728 kB
MemFree:          166568 kB
MemAvailable:     100656 kB
Buffers:            6520 kB
Cached:            27416 kB
SwapCached:          300 kB
Active:            17904 kB
Inactive:          16076 kB
Active(anon):        360 kB
Inactive(anon):      212 kB
Active(file):      17544 kB
Inactive(file):    15864 kB
Unevictable:          32 kB
Mlocked:              32 kB
SwapTotal:      33452028 kB
SwapFree:       33317332 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:           484 kB
Mapped:            23276 kB
Shmem:               144 kB
Slab:             559236 kB
SReclaimable:      60016 kB
SUnreclaim:       499220 kB
KernelStack:        8864 kB
PageTables:        10132 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    49866392 kB
Committed_AS:    1143048 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      358064 kB
VmallocChunk:   34342563088 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:    32637928 kB
DirectMap2M:    18446744073709318144 kB
DirectMap1G:     3145728 kB

Update2 Just ran things again and had the cmd "free" get captured every 15 seconds.. I watched the free column go down as the program ran until it reached a very low value (about 190000) and when it reached that level, the program hung and everything started going very slowly.. I ctrl-c'ed the program after it hung for a bit and eventually the terminal responded and took me back to a prompt.. However, "free" still reports about 190000 in the free column and even general use (just typing in a terminal) is very slow - no program is currently running. Looking at /proc/meminfo, the "DirectMap2M" field is crazy again.. I did capture the contents of /proc/meminfo regularly as well and can take a look at how things changed over time.

FYI: here's the output of "free" command when things hung up:

             total       used       free     shared    buffers     cached
Mem:      32828728   32636496     192232          4       7368      22972
-/+ buffers/cache:   32606156     222572
Swap:     33452028     205160   33246868

Here's a plot of the DirectMap2M value from /proc/meminfo over time. After the right-most point in the plot, it went to the ridiculous huge value - looks like an underflow.. A little bit of googling found others with an underflow issue here.. I don't know what DirectMap2M represents though..

enter image description here

Update 3: Still fighting this.. Some recently discovered info to add:

We've got it down as much as we could to this:

#include "cublas_v2.h"
int main() {
  cublasHandle_t handle;
  cublasCreate(&handle);
  cublasDestroy(handle);
  return 0;
}

Every time we run that on the Dell T630 with the NVidia K40, we see DirectMap2M go down. If we do it enough, we see the underflow issue and the machine has to be rebooted.. We also have a Dell R730 with an NVidia K80 that shows the same behavior..

Interestingly, we have yet another computer (its a laptop with an NVidia GTX980M) running the same Ubuntu kernel and we do not observe this behavior when we run the above..

  • 2
    You say "beefy servers" but do not describe at all what CPU or how much RAM they have. Also, Out of Memory is not necessarily the cause. The kernel will start killing processes when you hit OOM. Also you haven't really asked a question here. You've written a lengthy post to incite discussion, not asked something for which there can be a single direct answer applied. – dobey Feb 9 '16 at 16:25
  • Ok, sorry - I've updated the question to include some additional info. Was trying to avoid cluttering the question with info that might not add much.. My fault. When you say "OOM is not necessarily the cause" but then that the kernel will start killing processes when you hit OOM, it seems like you're saying OOM is the cause.. I'm not fully understanding. – daroo Feb 9 '16 at 16:56
  • The kernel OOM killer will try to select the most appropriate process to kill based on the memory used and how active the process is. If you can see what the OOM killer is killing then that's the mostly likely memory sucking problematic candidate process – Colin Ian King Feb 15 '16 at 16:43
  • DirectMap2M: the kernel can do linear (that is a "direct") mappings of physical memory. This is different from the virtual mappings allocated to each user process. The kernel attempts to use the largest pages where possible to reduce TLB (translation lookup buffer) pressure. The stats do look stupidly wrong, 18446744073709318144 is FFFFFFFFFFFC7000 in hex, showing it is an underflow issue with the stats – Colin Ian King Feb 15 '16 at 16:49
1

A good place to start is to track the statistics in /proc/meminfo, this has a considerable amount of detail on global memory utilisation. I suggest capturing the output from /proc/meminfo periodically (say every 30 minutes or so) and one can then examine this to see where the memory allocation growth is occurring. From that, you will have at least some idea on where to look next.

  • Great thanks - I'll try to monitor that when I start up the offending process here shortly.. – daroo Feb 10 '16 at 20:53
  • added the contents of /proc/meminfo to the question after it started acting problematic (the python program had crashed - the remote terminals were still responding, but very slow, and there was no response to mouse or keyboard at the physical console..) I tried "sudo /sbin/reboot" from the remote terminal - got the "machine is rebooting" message, but never rebooted - had to hit power button. I'm not sure what to make of the values I see in the meminfo dump above.. – daroo Feb 12 '16 at 15:36
  • I expect your machine is swapping itself like crazy. You could turn off overcommit memory and that way any process that's overly greedy will get stopped once it over allocates. Use: echo 2 | sudo tee /proc/sys/vm/overcommit_memory – Colin Ian King Feb 15 '16 at 18:32
  • I think that's right, but nut sure why.. I took the outer script out of the picture, and literally ran the command that was being called from inside the script by hand on the command line a bunch of times. There is nothing persistently running to "hold onto" memory, and even in this case, I see the value reported for DirectMap2M go down each time I run it and after it underflows the machine hangs. I still don't understand how these resources are being used up even after all processing completes. – daroo Feb 18 '16 at 12:12
  • Added "Update 3" to the question above. Narrowed it down to a call to cublasCreate(...). Still not sure if its a cublas or Ubuntu thing that is causing the problem, or if the underflow is just a display issue versus being the root of the issue. – daroo Feb 18 '16 at 18:34
1

We experienced the same problem with CUDA jobs running on Debian Jessie machines with GTX 970 and GTX 980Ti GPUs. Your test case also caused our machines to run out of memory within minutes.

What eventually fixed this annoying behavior for us was to install the most current beta driver from nvidia - version 364.12 at the time of writing. It appears to be independent of the Linux kernel (we tried several) and the CUDA version (we also tried several). This appears to have been a bug in the nvidia driver itself that was fixed only recently.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.