19

I seem to have a larger memory leak on my current ubuntu System

After reporting strange Eclipse memory errors ( https://askubuntu.com/questions/148998/eclipse-constant-different-out-of-memory-errors ) I started to get 'Not enough Memory' error messages in my console today - while doing simple tasks like typing in sudo -s - or even - free -m

Typing in 'free -m' repeadetly showed me how my RAM quickly goes up from 700M to 900M, growing up to the size of 2000M in a few seconds (after freeing up memory with echo 3 > /proc/sys/vm/drop_caches )

Eclipse isnt the cause, I completly killed the process and the ram still was going up. Is there any way to detect where the leak is coming from? I cant even update my system anymore, since apt-get update fails (probably because it's out of memory)

Using Ubuntu 11.10

Improve this question
2
  • I am VERY happy I am not crazy. I've had the same problem since upgrading to 13.10, but i remember having it with 11.10. The question is: Are you using CrashPlan? I seem to have it narrowed down to that, i just don't know how to fix it. I've tried the memory tweaks, but it is not working. I hope it gives you some clues
    – semi-newbie
    Nov 30, 2013 at 17:59
  • There's no point in forcing the kernel to drop caches. They will be flushed and their space reclaimed anyway as soon as more physical memory is required otherwise. Force-flushing them most likely even detrimental to overall performance, since uncached objects need to be retrieved from much slower secondary storage. Free main memory is by no means a good thing. It's either a sign of bad cache management or very light usage.
    – David Foerster
    Nov 30, 2014 at 23:53

6 Answers 6

Reset to default
16

First, make sure to have a temp folder available which has enough free space. The following commands create dumps which can be several GBs in size.

You can create a new tmp folder using the following command. You may want to change /tmp to a different filesystem with enough space

TMPDIR=$(mktemp -d -t -p /tmp)

Steps to find Memory Leak

  1. Find out the PID of the process which causing memory leak (you can also use e.g. htop if available) and store it in a variable called pid

    ps -aux

  2. Given that the PID is available in the variable pid, you can capture the memory consumption using /proc/$pid/smaps and save into some file like beforeMemInc.txt.

    cat /proc/$pid/smaps > $TMPDIR/beforeMemInc.txt
  3. Wait some time for memory consumption to increase.

  4. Capture /proc/$pid/smaps again and save it as afterMemInc.txt

    cat /proc/$pid/smaps > $TMPDIR/afterMemInc.txt

  5. Find the difference between first smaps and 2nd smaps, e. g. with

    diff -u $TMPDIR/beforeMemInc.txt $TMPDIR/afterMemInc.txt

  6. Note down the address range where memory got increased, for example:

       beforeMemInc.txt            afterMemInc.txt
---------------------------------------------------
2b3289290000-2b3289343000   2b3289290000-2b3289343000  #ADDRESS
Shared_Clean:    0 kB       Shared_Clean:    0 kB          
Shared_Dirty:    0 kB       Shared_Dirty:    0 kB
Private_Clean:   0 kB       Private_Clean:   0 kB
Private_Dirty:  28 kB       Private_Dirty:  36 kB  
Referenced:     28 kB       Referenced:     36 kB
Anonymous:      28 kB       Anonymous:      36 kB  #INCREASE MEM
AnonHugePages:   0 kB       AnonHugePages:   0 kB
Swap:            0 kB       Swap:            0 kB
KernelPageSize:  4 kB       KernelPageSize:  4 kB
MMUPageSize:     4 kB       MMUPageSize:     4 kB
Locked:          0 kB       Locked:          0 kB
VmFlags: rd wr mr mw me ac  VmFlags: rd wr mr mw me ac

  7. Use GDB to dump memory on running process or get the coredump using 

    gcore -o $TMPDIR/process $PID

  8. I used gdb on the running process to dump the memory to some file.

    cd $TMPDIR
gdb -p $pid
dump memory memory.dump 0x2b3289290000 0x2b3289343000

  9. Now, use strings command or hexdump -C to print the memory.dump

    strings memory.dump

    From this you get readable information which helps you locate those strings in your source code.

  10. Analyse your source to find the leak.
Improve this answer
2
  • I am in a Docker container, getting a Permission denied error when running cat /proc/2882/smaps > /tmp/before.txt on step 2. What did I do wrong?
    – Devy
    Apr 2, 2018 at 21:15
  • This method worked for me up until step 9. The leak in my case didn't contain any readable or recognisable data. How can I get the symbol names of the data that was leaked?
    – Craig
    Jan 16, 2020 at 10:22
12

The drop_cache trick will not free memory, it will reset the cache. Uses ps command if you want to identify which processes uses the more memory.

For instance to monitor the list of the top 15 of resident memory users.

$ watch "ps --sort -rss -eo pid,pmem,rss,vsz,comm | head -16"
  PID %MEM   RSS    VSZ COMMAND
 2590 13.4 136892 825000 firefox
 1743 10.7 109020 300780 Xorg
 2067  8.5 86764 1118140 unity-2d-shell
 3307  4.1 42560 627780 unity-2d-spread
 2068  2.9 29904 617644 unity-2d-panel
 2092  2.5 25524 1291204 nautilus
 2457  1.9 20292 530276 gnome-terminal
 2351  1.9 20016 821488 unity-scope-vid
 2161  1.9 19476 531968 unity-panel-ser
 2034  1.7 18256 759716 gnome-settings-
 2074  1.5 16176 518016 nm-applet
 2273  1.5 15452 580416 unity-lens-vide
 2051  1.4 15112 524260 metacity
 2395  1.2 12836 407336 update-notifi

You could check also the shared memory reservation but you will only know who is the owner of the segments.

Pmap allocation:

$ ls -l /run/shm
total 272
-r-------- 1 ed      ed      67108904 Nov 29 18:17 pulse-shm-1884617860
-r-------- 1 lightdm lightdm 67108904 Nov 29 18:11 pulse-shm-2352897759
-r-------- 1 ed      ed      67108904 Nov 29 18:12 pulse-shm-3444873503
-r-------- 1 ed      ed      67108904 Nov 29 18:12 pulse-shm-3485341848
-r-------- 1 lightdm lightdm 67108904 Nov 29 18:11 pulse-shm-535843976
-r-------- 1 ed      ed      67108904 Nov 29 19:12 pulse-shm-789046959
-r-------- 1 ed      ed      67108904 Nov 29 18:38 pulse-shm-863909656

$ df /run/shm 
Filesystem     1K-blocks  Used Available Use% Mounted on
none              509332   272    509060   1% /run/shm

note that reserved allocations are much higher than real the allocated pages (df 'used')

System V allocations :

$ ipcs -m 

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x00000000 294912     ed         700        122880     2          dest         
0x00000000 327681     ed         700        4823040    2          dest         
0x00000000 491522     ed         600        393216     2          dest         
0x00000000 589827     ed         700        4578120    2          dest         
0x00000000 425988     ed         700        27852      2          dest         
0x00000000 458757     ed         600        393216     2          dest

Edit: Need to pass --sort -rss to ps to get the processes with the most memory usage, otherwise the process list is sorted increasing numerical and gives the processes with least memory usage.

Improve this answer
9

memprof is a tool for profiling memory usage and finding memory leaks. It can generate a profile how much memory was allocated by each function in your program. Also, it can scan memory and find blocks that you’ve allocated but are no longer referenced anywhere.

memprof works by pre-loading a library to override the C library’s memory allocation functions and does not require you to recompile your program.

memprof

Source: Ubuntu Manual

Improve this answer
0
5

I have an older machine that I use that constantly spits out memory leak messages:

root@:~# free -m
             total       used       free     shared    buffers     cached
Mem:          1898       1523        374        131         32        588
-/+ buffers/cache:        902        995
Swap:         1942        480       1462

My script:

sync; sudo echo 3 > /proc/sys/vm/drop_caches

Named it cache.sh

root@~# ./cache.sh
root@~# free -m
             total       used       free     shared    buffers     cached
Mem:          1898       1106        791        126          1        207
-/+ buffers/cache:        897       1000
Swap:         1942        480       1462

You can see I was down to 374 MB, ran the sync; sudo echo 3 > /proc/sys/vm/drop_caches and gained 417 MB back. One can cron it to run every 5 minutes or just have a terminal open and run it when you see slow performance. Yes, I need to add memory to the machine...

Improve this answer
4
  • Formatting seems to be an issue, not sure how to fix
    – Warpig
    Nov 30, 2014 at 23:23
  • 1
    Use the edit link beneath your post. There are a formatting toolbar and an orange question mark above the text area linking to the Markdown formatting help.
    – David Foerster
    Nov 30, 2014 at 23:43
  • Please have a look at my recent comment on the question. I'm convinced that the idea to free main memory by flushing and dropping caches is misguided and I know I'm not alone with that conclusion.
    – David Foerster
    Nov 30, 2014 at 23:58
  • Many thanks, David... I totally agree of flushing / dropping cache is misguided... But something is getting hung up and making the machine freeze / lockup... Just baffled as to what it is, thinking its a firefox issue...
    – Warpig
    Dec 1, 2014 at 0:24
5

memstat is also a good tool that will show the amount of memory used by each block as well as how much memory is used by loaded libraries. Not the best tool but is worth using to gather details and statistics.

memstat -w -p pid is a good command to use.

Improve this answer
1
  • 1
    link is broken, I think this one is good
    – vladkras
    Oct 13, 2016 at 13:38
1

I had a similar problem but with a very weird solution.

For some unknown reason I had a mail server on my laptop setup and running.I don't know why I had it...However I shut its service down and it turned out that that software on my laptop was under a ddos attack.After that everything was normal.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .