36

I'm having a problem with Ubuntu 12.04 LTS. This is the second time I've encountered this problem in the last 3 weeks. The first time is described in this closed question on StackOverflow. The TL;DR version is I managed to use all the inodes on a 450G ext4 system compiling and building the Android stack less than 20 times.

I thought I was solving the problem by reformatting the disk as XFS so the inode storage could grow.

This morning after doing a build overnight, I'm down to less than 1GB of free space. There is nothing on this machine other than what is needed to build Android. I've done 5 total builds on the platform sources. The build creates a bunch of files, then I delete them shortly after with make clean. I'm not really less than 1GB but the tools are reporting it that way. I deleted a bunch of temporary files and had about 40GB "freed". A couple hours later, just idling, I was back to less than 1GB free.

Running Ubuntu from a flash drive returns the following for the partition...

$ df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/sda5      468521456 468255460    265996 100% /media/f71c77eb-b4cc-

$ df -i
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/sda5      1691760 624214 1067546   37% /media/f71c77eb-b4cc-

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda5       447G  447G  260M 100% /media/f71c77eb-b4cc-

Here's is the evidence that something is amiss. When I run du (with and without --apparent-size) or the visual Disk Usage Analyzer, I show that I'm really only using about 35GB or so. 98.7% of the used space is in /home/eric, but du doesn't add up to that. The discrepancy is between /home/eric and /home/eric/android

enter image description here

I've read through the relevant questions here and on SO and they usually suggest it's deleted files held by open processes. I rebooted into a flash drive to run this test, so it shouldn't be open files. FWIW, /tmp is empty.

Is there a tool I can install on a flash drive to recover the 'lost' space? I can try to free up memory on the system and run it there, but I'm assuming it's better to do this from a flash drive.

Should I be configuring this system in a different way? I would rather not do another wipe and install, but I need a sustainable Android build system.

FOLLOWUP - I had to nuke the installation last week and re-installed 12.04 in order to get work done. As I go through Android builds again this week, I will keep a watchful eye over the disk usage and give information here as I learn more.

Thanks

6
  • 1
    Have you checked Why df and du show different output?
    – jokerdino
    Feb 18, 2013 at 17:02
  • @jokerdino Yes, I followed the directions in that thread and some of the other similar threads. I restarted the system as well as restarting from a flash drive. After the restart the system still shows the whole disk as occupied. Thanks. Feb 19, 2013 at 2:41
  • File data in Linux is reference counted hence you don't need to restart as often as in Windows. Perhaps you kept the emulator running while overwriting its data files under its back?
    – aquaherd
    Feb 24, 2013 at 17:11
  • @EricCloninger I have to add my self to the list of users how happened the same issue. This is my question using 12.10 and this is the question of a 11.04 user.
    – Lucio
    Feb 25, 2013 at 1:16
  • Also, copy and paste the output of ls -lah ~ in paste.ubuntu.com and link it in your question.
    – Lucio
    Feb 25, 2013 at 1:39

9 Answers 9

23

On Oracle Linux it happens when you have (a lot of/large) files which are deleted but still open by a running process. Then stop of the processes or reboot of the machine helps.

2
  • Worked for me on a server hugging the log files. Thanks!
    – miccet
    Feb 10, 2015 at 11:42
  • 19
    Use lsof +L1, to check the process id, and kill it.
    – Jeff Tian
    Oct 31, 2016 at 9:57
4

I recently ran into this, and in my case a fsck needed to be run.

I did touch /forcefsck && reboot and after a few minutes, the server was back online and suddenly my missing 6 GB freed up.

2

This is frequently caused by files within a directory, which also has a different filesystem mounted on it. A typical fix is to boot with a rescue disk, or in single user mode, and empty the directories, having verified that they are not being used as mount points (cat /proc/mounts or df -h).

1
  • This was my issue, somehow a docker container had mounted a file before the NFS share that it should be in was mounted, and had written sever gigabytes of recording to the root filesystem.
    – SilentVoid
    Sep 24, 2021 at 20:51
1

Before you go to far.....take the system down to single user mode and do a FULL fsck (I really mean full fsck -f /dev/sda5) of the filesystem and see what it shows. You might find the space as portions of problem areas on your disk or mismatch between what is allocated and what is present on the disk.

1
  • sudo fsck -f /dev/sda5 from a flash drive did nothing as the drive is formatted XFS. It advised me to use xfs_check. xfs_check /dev/sda5 returned no results. xfs_repair /dev/sda5 ran through its repairs but didn't report anything unusual. After both, I'm still 99% full. Thx Feb 21, 2013 at 17:57
1

We can do a test, du says you have 10GB free space, while df says 300 MB, can you write a file (or several files) with size of, say, 2 GB? If you can, that means that df is just simply wrong (and there is actually no problem of 'lost space'). If not, then du is wrong (which will be interesting).

1

I have not found the cause of this problem, but it is unique to ubuntu 12.04.

Just setting up a new server, I started with Ubuntu 12.04 and ran into this; du showed about 111 GiB usage, while df was around 170 GiB.

Booting using systemrescuecd 3.3.0 and checking again showed a difference of less than 1 GiB.

Leaving the partition and file system (it is ext4) unchanged, I moved the ubuntu directories out of the way and installed Debian 7.0. Again, the difference between du and df was < 1 GiB.

With ubuntu 10.04, on the same partition and ext4 fs:

From df -m /,

Filesystem           1M-blocks      Used Available Use% Mounted on
/dev/sda2              2814679    407164   2264538  16% /

and from du -mx,

tail -1 /root/diskuse 
406920  .

which is close enough.

0

The image you posted is telling you where the space is being used: /home/eric. It seems you have a very large file in there hogging all the space, or possibly a large number of smaller files. Open your home directory, make sure to show hidden files (Ctrl+H in Nautilus ), and sort by file size.

0

Have you tried to restart your system see if the space got reclaimed?

One possible scenario is that if you (or your system) deleted file that took a lot of space but these files have an open pointer, the disk space will not be reclaim until all the pointers are closed, hence why it "looks" like it has space, but the filesystem actually disagrees, and if you run out of disk space, the filesystem will refuse to write more files to disk until all the pointers are closed.

There isn't a way that works cleanly for all, it's really case to case, but in a case I had, syslog file was like 41Go, and even after deleting it, the space was not reclaimed until I closed the process that had an open pointer to it, in my case Google Cloud Ops Agent Fluent-bit client.

In my case, I had to use $ lsof command to find out what service had a hold over it, then use $ service or sudo systemctl restart [SERVICE_NAME]. Worked as expected. (the other nuke option is to restart, but that's an ugly way)

-1

Files in trash usually clog up the disk. It is easy to see them using "Disk usage analyser". The files can be deleted by searching for "Trash". There is the option to delete the contents of the "Trash". There is also the option to delete "temp" files. One can also set option to automatically delete "Trash" at intervals.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .