0

So, I've been hunting for the answer to this, but can't find anything. One day I magically ran out of space on my root filesystem. 0 bytes free. I deleted some old VMs to give me space back so the system would actually run. I've run Disk Usage Analyzer, k4DirStat and df / du. All as SU and the math doesn't add up. It's not the 5% for Ext4 either.

The drive is a Samsung EVO 840. Ext4. Ubuntu 16.04 LTS.

Here are the results of du and df:

XXX@YYY:/$ sudo du -chsx /
103G    /
103G    total

XXX@YYY:/$ sudo df /
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/sdf1      231977984 201086936  19084124  92% /

So: 231G - 103G = 19G?

Stumped and thanks for your help.

Solution: @steeldriver had the right idea. I have two external drives connected for backup cron jobs. I unmounted them. Disconnected them. Viola, there were still files in the mount points. I guess one of them became unmounted at some point and the backup job filled the root filesystem. I guess I'll need to change the script to check for the existence of the external drives before syncing.

Thanks for the quick responses - this community is fantastic.

4
  • The root disk can fill up if something writes to a mountpoint while the device that's supposed to be mounted there isn't actually mounted Sep 25, 2018 at 2:20
  • Well, you're also looking at different units. Output of du is rounded to G, the values of df are in 1024 blocks (k). Sep 25, 2018 at 3:33
  • Some discrepancy is to be expected as du gives the file size in bytes, and df the actual blocks allocated on disc (a 1-byte file takes up one block which is usually 4096 bytes). However the 98G missing would then need millions of tiny files. Could it be that the file system was formatted with a 1M block size, and there are lots of tiny files? What is the output of sudo dumpe2fs -h /dev/sdf1 | grep 'Block size'?
    – zwets
    Sep 25, 2018 at 6:57
  • Yes, dumpe2fs -h /dev/sda1 would be nice to add, full and unfiltered. Also please add sudo du --summarize --bytes --one-file-system / and df --block-size=1 --local outputs Sep 25, 2018 at 7:57

1 Answer 1

1

Linux will not remove files from the file system, which are still opened by an application. In your case this could be some large VM images.

Example:

Create a large file check the disk usage. Also use sync to avoid caching effects:

$ sudo sync; df -h /dev/sda1
  Filesystem      Size  Used Avail Use% Mounted on
  /dev/sda1       886G  278G  564G  33% /
$ dd if=/dev/zero of=large count=1024 bs=10M
$ df -h /dev/sda1
  Filesystem      Size  Used Avail Use% Mounted on
  /dev/sda1       886G  288G  554G  35% /

Now open the file in a other shell for example with tail and leave it open:

$ tail -f large

Go back to the first shell and delete the file and check the file system size:

$ rm large
$ sudo sync; df -h /dev/sda1
  Filesystem      Size  Used Avail Use% Mounted on
  /dev/sda1       886G  288G  554G  35% /
$ lsof /dev/sda1 | grep large
tail      15255 username    3r   REG    8,1 10737418240  5520283 /some/path/large (deleted)

lsof shows, that the file was delete, but it is still used by some processes. Now go back to the tail shell and terminated it with Ctrl-C:

$ tail -f large 
^C
$ sudo sync; df -h /dev/sda1
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       886G  278G  564G  33% /

Back to normal. So you need to find the opened files and terminated the processes holding the file handle.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.