I am running Ubuntu 14.04LTS on a Dell PowerEdge R730 with 32 GB of RAM and an Nvidia K80 GPU.. I have a python script that runs for a long time, but really just makes a bunch of relatively short-lived system calls to other programs (some of which use CUDA extensively in case that matters).. Other than the overarching python script, the individual system calls only run for about 45 seconds a piece, and the python script itself doesn't "maintain" anything - it just does the system calls and iterates - not any storage of results, etc..
As I watch the program run and monitor memory usage with "top", I see the reported "free" memory go down over time.. I understand this isn't usually a concern due to caching being employed. Eventually, though, the machine hangs and doesn't respond in any way (i.e. no mouse cursor movement on console, no response from remote terminals, etc).. When this happens, I have to hard reboot the system, and things are back to normal, and I can pick up where I left off, until it hangs again..
Another point of interest - after enough reboot-and-restarts the program finally completes.. Again, after this, there is nothing of significance running on the system, however, the gnome-system-monitor and top both report almost full memory usage, and trying to run other (minimal) commands are often "Killed" by the kernel.. Checking the system logs, it reports "out of memory" which seems to indicate that the memory being used is not cache but memory that is actually "claimed". The question is - by who? Checking all the memory utilities that people have mentioned on this and other forums, no processes claim to be using much memory..
I've read reports of kernel memory leaks, and that's what I want to claim, but it just seems unlikely..
Question: How can I determine what is claiming all the memory? I'd like to be able to determine if its in the kernel or if its some process that running.
Supporting Infomation:
uname -a:
Linux machinename 3.19.0-47-generic #53~14.04.1-Ubuntu SMP Mon Jan 18 16:09:14 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
lscpu:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Stepping: 2
CPU MHz: 1200.351
BogoMIPS: 5993.09
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 10240K
NUMA node0 CPU(s): 0-7
Prior Reading and understanding: I've read through a ton of posts about understanding Linux's memory usage and reporting.. I understand that when programs like top report very little memory "free" it is not necessarily a problem because much of the "used" memory is cached, and actually having your RAM full of cached stuff is good.. I believe, however, that this is not the issue that I am seeing, because if it were cache it seems like programs would be able to make use of it. The fact that the kernel is stepping in and killing new processes and dmesg is reporting the system is "out of memory" seems to indicate memory is being occupied in a non-cache way, but it doesn't seem to be being reported in any memory analysis tool I have tried..
Update: Based on an answer below, I looked at /proc/meminfo when things were starting to go poorly, and while I don't know what all these mean, there are a few that seem .. suspect.. "DirectMap2M" seems way problematic and "VmallocChunk" too, although less so...
> cat /proc/meminfo
MemTotal: 32828728 kB
MemFree: 166568 kB
MemAvailable: 100656 kB
Buffers: 6520 kB
Cached: 27416 kB
SwapCached: 300 kB
Active: 17904 kB
Inactive: 16076 kB
Active(anon): 360 kB
Inactive(anon): 212 kB
Active(file): 17544 kB
Inactive(file): 15864 kB
Unevictable: 32 kB
Mlocked: 32 kB
SwapTotal: 33452028 kB
SwapFree: 33317332 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 484 kB
Mapped: 23276 kB
Shmem: 144 kB
Slab: 559236 kB
SReclaimable: 60016 kB
SUnreclaim: 499220 kB
KernelStack: 8864 kB
PageTables: 10132 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 49866392 kB
Committed_AS: 1143048 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 358064 kB
VmallocChunk: 34342563088 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 32637928 kB
DirectMap2M: 18446744073709318144 kB
DirectMap1G: 3145728 kB
Update2 Just ran things again and had the cmd "free" get captured every 15 seconds.. I watched the free column go down as the program ran until it reached a very low value (about 190000) and when it reached that level, the program hung and everything started going very slowly.. I ctrl-c'ed the program after it hung for a bit and eventually the terminal responded and took me back to a prompt.. However, "free" still reports about 190000 in the free column and even general use (just typing in a terminal) is very slow - no program is currently running. Looking at /proc/meminfo, the "DirectMap2M" field is crazy again.. I did capture the contents of /proc/meminfo regularly as well and can take a look at how things changed over time.
FYI: here's the output of "free" command when things hung up:
total used free shared buffers cached
Mem: 32828728 32636496 192232 4 7368 22972
-/+ buffers/cache: 32606156 222572
Swap: 33452028 205160 33246868
Here's a plot of the DirectMap2M value from /proc/meminfo over time. After the right-most point in the plot, it went to the ridiculous huge value - looks like an underflow.. A little bit of googling found others with an underflow issue here.. I don't know what DirectMap2M represents though..
Update 3: Still fighting this.. Some recently discovered info to add:
We've got it down as much as we could to this:
#include "cublas_v2.h"
int main() {
cublasHandle_t handle;
cublasCreate(&handle);
cublasDestroy(handle);
return 0;
}
Every time we run that on the Dell T630 with the NVidia K40, we see DirectMap2M go down. If we do it enough, we see the underflow issue and the machine has to be rebooted.. We also have a Dell R730 with an NVidia K80 that shows the same behavior..
Interestingly, we have yet another computer (its a laptop with an NVidia GTX980M) running the same Ubuntu kernel and we do not observe this behavior when we run the above..