35

So a client of mine got an email from Linode today saying their server was causing Linode's backup service to blow up. Why? Too many files. I laughed and then ran:

# df -ih
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/xvda        2.5M  2.4M   91K   97% /

Crap. 2.4million inodes in use. What the hell has been going on?!

I've looked for the obvious suspects (/var/{log,cache} and the directory where all the sites are hosted from) but I'm not finding anything really suspicious. Somewhere on this beast I'm certain there's a directory that contains a couple of million files.

For context one my my busy servers uses 200k inodes and my desktop (an old install with over 4TB of used storage) is only just over a million. There is a problem.

So my question is, how do I find where the problem is? Is there a du for inodes?

2

7 Answers 7

29

Check /lost+found in case there was a disk problem and a lot of junk ended up being detected as separate files, possibly wrongly.

Check iostat to see if some application is still producing files like crazy.

find / -xdev -type d -size +100k will tell you if there's a directory that uses more than 100kB of disk space. That would be a directory that contains a lot of files, or contained a lot of files in the past. You may want to adjust the size figure.

I don't think there's a combination of options to GNU du to make it count 1 per directory entry. You can do this by producing the list of files with find and doing a little bit of counting in awk. Here is a du for inodes. Minimally tested, doesn't try to cope with file names containing newlines.

#!/bin/sh
find "$@" -xdev -depth | awk '{
    depth = $0; gsub(/[^\/]/, "", depth); depth = length(depth);
    if (depth < previous_depth) {
       # A non-empty directory: its predecessor was one of its files
       total[depth] += total[previous_depth];
       print total[previous_depth] + 1, $0;
       total[previous_depth] = 0;
    }
    ++total[depth];
    previous_depth = depth;
}
END { print total[0], "total"; }'

Usage: du-inodes /. Prints a list of non-empty directories with the total count of entries in them and their subdirectories recursively. Redirect the output to a file and review it at your leisure. sort -k1nr <root.du-inodes | head will tell you the biggest offenders.

2
  • The script give errors: awk: line 2: find: regular expression compile failed (bad class -- [], [^] or [) [^ awk: line 2: syntax error at or near ] `/tmp/tmpw99dhs': Permission denied Jul 4, 2013 at 7:52
  • @RaduRădeanu Ah, I see, I used a gawk peculiarity that doesn't work in other versions. I've added a backslash which I think is necessary as per POSIX. Jul 4, 2013 at 8:08
15

You can check with this script:

#!/bin/bash

if [ $# -ne 1 ];then
  echo "Usage: `basename $0` DIRECTORY"
  exit 1
fi

echo "Wait a moment if you want a good top of the bushy folders..."

find "$@" -type d -print0 2>/dev/null | while IFS= read -r -d '' file; do 
    echo -e `ls -A "$file" 2>/dev/null | wc -l` "files in:\t $file"
done | sort -nr | head | awk '{print NR".", "\t", $0}'

exit 0

This prints the top 10 subdirectories by file count. If you want a top x, change head with head -n x, where x is a natural number bigger than 0.

For 100% sure results, run this script with root privileges:

top-bushy-folders

2
  • 2019: raised 10: read: Illegal option -d ... scrubbed the -d flag from read hoping nothing bad will happen. Will let you know when it finishes running ...
    – Williams
    Jun 5, 2019 at 0:07
  • If the file system is close to 100% full you don't want to create another temp file that makes the situation worst. That's wy I like Gille's answer better as it does not create a new file.
    – Guasqueño
    Oct 15, 2021 at 14:24
6

Often faster than find, if your locate database is up to date:

# locate '' | sed 's|/[^/]*$|/|g' | sort | uniq -c | sort -n | tee filesperdirectory.txt | tail

This dumps the entire locate database, strips off everything past the last '/' in the path, then the sort and "uniq -c" get you the number of files/directories per directory. "sort -n" piped to tail to get you the ten directories with the most things in them.

2
  • +1: using the locate database is a very nice idea! Mar 13, 2015 at 21:40
  • When you can't use locate for whatever reason, run a find /path/to/parent -xdev > filelist first, then direct sed to read input from that list.
    – gerrit
    Aug 16, 2018 at 11:16
4

a bit old thread but interesting so I suggest my solutions.

First uses few piped commands and it finds directories with over 1000 files inside:

find / -type d  |awk '{print "echo -n "$0" ---- ; ls -1 "$0" |wc -l "}'|bash |awk -F "----" '{if ($2>1000) print $1}'

Second is simple. It just try to find directories that have size over 4096B. Normally empty directory has 4096B on the ext4 filesystem i 6B on the xfs:

find / -type d -size +4096c

You can adjust it of course but I believe that it should work in most cases with such value.

1

Another suggest:

http://www.iasptk.com/20314-ubuntu-find-large-files-fast-from-command-line

Use these searches to find the largest files on your server.

Find files over 1GB

sudo find / -type f -size +1000000k -exec ls -lh {} \;

Find files over 100MB

sudo find / -type f -size +100000k -exec ls -lh {} \;

Find files over 10MB

sudo find / -type f -size +10000k -exec ls -lh {} \;

The first part is the find command using the "-size" flag to find files over different sizes measured in kilobytes.

The last bit on the end starting with "-exec" allows to specify a command we want to execute on each file we find. Here the "ls -lh" command to include all the information seeing when listing the contents of a directory. The h towards the end is especially helpful as it prints out the size of each file in a human readable format.

1
  • 2
    His problem is high inode usage, which points to many smaller files, not large ones.
    – UpTheCreek
    Jun 24, 2015 at 19:38
0

This worked for me when the other's failed on Android through the shell:

find / -type d -exec sh -c "fc=\$(find '{}' -type f | wc -l); echo -e \"\$fc\t{}\"" \; | sort -nr | head -n25
0

I like to use something like du --inodes -d 1 to find a directory that either recursively or directly contains a lot of files.

I also like this answer: https://unix.stackexchange.com/a/123052

For the lazy of us, here is the gist of it:

du --inodes -S | sort -rh | sed -n \
    '1,50{/^.\{71\}/s/^\(.\{30\}\).*\(.\{37\}\)$/\1...\2/;p}'

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .