6

my current system: 14.04.4 LTS (GNU/Linux 3.13.0-85-generic x86_64)

$ free
             total       used       free     shared    buffers     cached 
Mem:      16366288   16090588     275700    4047472    1019652  5253144
-/+ buffers/cache:    9817792    6548496 Swap:      7813116    7308592     504524



$ cat /proc/sys/fs/file-nr
1196103  0   1624594

For a couple of months (and a number of updates) I've got the problem that suddenly after some days of working with my box (and daily suspending it) memory usage and the number of allocated file descriptors shoots up.

All of a sudden the "committed" memory rises, up to around 120 - 160 GB(!). Normally this value sits around my physical RAM. Another symptom is that all my RAM + swap is used up. The result is an unusable system. For now I only could fix this situation by rebooting. Killing nearly all processes after having shut down X did not help to free up the committed value.

A strong indicator seems to be the unreasonably high value of the open files counter in /proc/sys/fs/file-nr -- this goes up above 2 million. Lately I tried to limit this to 1624594 but the only result is (of course) that I don't run out of memory but instead out of FDs.

Having Munin in place I can see a strong temporal correlation between FDs and memory usage.

I have some CIFS shares mounted and my user has its gvfsd-fuse. Root FS is ext4.

The results of lsof and traversing /proc/ list only about 10% of /proc/sys/fs/file-nr.

I suspected MTP to be the cause (when I plug in my Motorola phone) but recently this happened without having used MTP since the last reboot.

Sidenote: I've got radeon drivers for video:

ii  libdrm-radeon1:amd64                                        2.4.64-1~ubuntu14.04.1                        
ii  libdrm-radeon1:i386                                         2.4.64-1~ubuntu14.04.1                        
ii  radeontool                                                  1.6.3-1                                       
ii  xserver-xorg-video-radeon                                   1:7.3.0-1ubuntu3.1  

UPDATE

root:/proc$ (echo -n "0" ; for processid in [0-9]*; do echo -n "+$(ls /proc/$processid/fd/ | wc -l)"; done;echo)|bc
2597
root:/proc$ lsof -n|wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
84504
root:/proc$ sudo -u mdo lsof -n|wc -l
72170

How can I find out what's eating all these FDs and my memory?

3

This will show the number of open files per process ID:

cd /proc
for processid in [0-9]*
do
    echo "Process ID = $processid: $(ls /proc/$processid/fd/ | wc -l) file descriptors"
done

(save and execute with sudo).

Then you can track down what that process ID belongs to.

2
  • Thanks for your comment, up to now I wasn't able to find any listing that roughly reaches the number in /proc/sys/fs/file-nr
    – mdo
    Jun 27 '16 at 8:33
  • modified it a bit so the number of descriptors is a first col and then piped it into ... | sort -g -k 1,1. Shows the highest users of descriptors as latest
    – thi gg
    Sep 26 '19 at 11:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.