Partition is reported to be full, but I cannot "see" any culprit files

Question

So here's the deal:

--df reports 100% usage on the partition hosting my /home directories (and only those)

--Tools that use du (a la baobab or xdiskusage) report that the excessive space consumption is clearly related to my user.

--Tools that use du do not, however, seem to be able to locate any files that are the culprit. For example, xdiskusage shows my profile taking up > 260GB but only about half that amount of data actually in files (the rest is shown as just empty space).

--Running du on CLI just shows what I've described above: my user's ~/ folder is consuming like twice as much disk space than it actually has files inside of it.

And my favorite:

--This appears to be correlated with uptime, as though some process is slowly filling the partition with data until the drive is full. And when I reboot -- pooof! tons of free space. Is there an introspection tool that'd let me ascertain, e.g., what process is writing to disk at any given time?

Any suggestions?

Answer 1

I don't know what is causing the behavior you're observing, but I can imagine how it is happening:

Due to the way Unix filesystem works, it is possible for an application to open a file, then unlink it (i.e. delete the file name entry from its parent directory) - the file handle still remains valid though there is no file name associated with it, and the application is still able to write to the file, because the handle is pointing to the inode, not to the filename.

Such a file exists until the application closes it. My understanding is that this is a common way to create a "private" temp file for an application.

To find such files, use

lsof +L

which will show you the number of links to each open file (NLINK column) - the unlinked files have 0 there. Here's how an unlinked file looks:

COMMAND     PID       USER   FD      TYPE             DEVICE SIZE/OFF NLINK       NODE NAME
...
mysqld     1126      mysql   11u      REG                8,1        0     0   33603687 /tmp/ibMlwWQA (deleted)

Answer 2

The curious part is how this changes with time, and then goes away when you log out. More about that in a minute.

How are you running du in a way that accounts for hidden files in your home directory? For example compare the output of the following two commands, run from your home directory:

(home)$ du -csxk *
(home)$ du -csxk .

The "dot" form should always be bigger. A lot bigger, and you know that your extra space is in a hidden file.

try the following, again from $HOME:

(home)$ find . -maxdepth 1 -print0 | xargs -0 du -cskx --exclude .gvfs | sort -rn

(I exclude .gvfs because that's where gnome mounts remote file systems.)

That should return a list of all directories, including the hidden ones, sorted descending by size. A total line and your home (.) should be at the top, and because you're doing a grand total, the total line should be roughly twice your home size.

From here, you can start drilling into large directories (my .thunderbird dir is big) looking for culprits.

Now, about the time-correlated part of this. First, let's see if you can see the sizes change as you're logged in:

(home)$ watch -n 10 'find . -maxdepth 1 -print0 | xargs -0 du -csxk --exclude .gvfs | sort -rn'

That will run the command every 10 seconds, so you can hopefully see things change over time. The next thing to do is trying to see the storage come back on logout. Make a second admin user (you need sudo privileges) and login as that user via a remote computer or something. As that user, sudo to root, and cd into your home and run the last command again, watching for something to change as you logout and login.