91

When running tail -f filename, I got the following message:

tail: inotify cannot be used, reverting to polling: Too many open files

Is that a potential problem?

How do I diagnose what's responsible for all the open files? I have a list of suspect processes, but if they don't turn out to be the culprits, instructions that don't rely on knowing which process to check would be useful.

Improve this question
4
  • 1
    Have you increased the number of file descriptors available via ulimit?
    – Ignacio Vazquez-Abrams
    Commented Aug 28, 2012 at 3:09
  • 2
    @IgnacioVazquez-Abrams That may be helpful to other users, but to me it'd feel like treating the symptom rather than the disease.
    – Andrew Grimm
    Commented Aug 28, 2012 at 3:13
  • While you're not wrong, sometimes apps have legitimate reasons for having many files open.
    – Ignacio Vazquez-Abrams
    Commented Aug 28, 2012 at 3:14
  • Related: Ubuntu - too many open files?
    – kenorb
    Commented Feb 3, 2019 at 0:08

3 Answers 3

Reset to default
86

You can use lsof to understand who's opening so many files. Usually it's a (web)server that opens so many files, but lsof will surely help you identify the cause.

Once you understand who's the bad guy you can

  • kill the process/stop the program
  • raise the ulimit

If output from lsof is quite huge try redirecting it to a file and then open the file

Example (you might have to Ctrl+C the first command)

lsof > ~/Desktop/lsof.log
cat ~/Desktop/lsof.log | awk '{ print $2 " " $1; }' | sort -rn | uniq -c | sort -rn | head -20
vim ~/Desktop/lsof.log
Improve this answer
10
  • 49
    For the lazy: lsof | awk '{ print $2; }' | uniq -c | sort -rn | head
    – itsadok
    Commented Nov 27, 2012 at 6:10
  • 1
    I got the same error and using ulimit doesn't work. The tail -F command still returns the error. I increased the limit from 1024 to 3000 so I would imagine that I have enough room by then... I guess I'd have to reboot!
    – Alexis Wilke
    Commented Jan 26, 2013 at 11:41
  • 18
    I found itsadok's line helpful, but I think you should sort first (because uniq only works with adjacent lines), run uniq, then sort again. So lsof | awk '{ print $2; }' | sort -rn | uniq -c | sort -rn | head.
    – Tyler Collier
    Commented Feb 2, 2013 at 0:45
  • 27
    Sorting and counting up the most files open is definitely the best thing to do. Show process name as well as pid: '''lsof | awk '{ print $2 " " $1; }' | sort -rn | uniq -c | sort -rn | head -20'''
    – gaoithe
    Commented Dec 1, 2014 at 11:34
  • 2
    @gaoithe if you put that into an answer I will gladly upvote it :)
    – Matt Ball
    Commented Jul 27, 2015 at 17:56
35

In case anyone else needs it...

ulimit -a

Will display all current limits. Specifically ulimit -n 70000 will set the file descriptor limit.

Also...

cat /proc/sys/fs/file-max

Will display/set the kernel limit if edited.

sudo echo 200000 > /proc/sys/fs/file-max

A much more detailed explanation can be found at...

How do I increase the open files limit for a non-root user?

Improve this answer
2
  • 4
    ulimits are there so you can keep tight control over resource use in your system. It is best to increase them individually for processes you know will need to use many file descriptors. By keeping them tight in testing you will find processes which might steadily leak file descriptors over time. Note also that file handles are used for any device access in unix/linux. e.g. every network socket open by a process uses a file handle. That explains why you can hit the "too many open files" in case of regular file-system files as well as any device files such as network connections.
    – gaoithe
    Commented Dec 1, 2014 at 11:51
  • "Too many files" (EMFILE) has nothing to do with /proc/sys/fs/file-max. The english translation for that error "Too many files for systerm" (ENFILE), indicating the system-wide limit. "Too many files" always refers to the per-process file descriptor limit.
    – Remember Monica
    Commented Sep 10, 2021 at 2:00
8

While ulimit can be used to determine how many files are allowed to be open per process you may want to find the culprit.

@itsadok @Tyler Collier @gaoithe in comments to other answers highlight that sorting and counting which process has the most files open is the best thing to do here:

sudo lsof | head -1 | awk '{ print "COUNT " $2 " " $1; }' && sudo lsof +c 0 | awk '{ print $2 " " $1; }' | sort -rn | uniq -c | sort -rn | head -20

Above command:

  • Gives output a header
  • lsof gives list of open files
  • +c 0 option instructs lsof to print full command
  • awk '{ print $2 " " $1; }' prints PID and COMMAND column in results
  • sort -rn sorts results so identical entries are next to each other (needed for uniq -c to work properly)
  • uniq -c counts files open by PID/command
  • sort -rn sorts results by count
  • head -20 shows top 20 files open by PID/command

Note: This will count includes "files" that don't count towards limits

You may want to investigate even further by looking at limits for a PID, # of files open for a specific PID, and limit lsof to only count files that count towards limit - see https://serverfault.com/a/964752/152562.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .