Utility says there's no more free disk space, when there's plenty

Question

Somehow my system is in a pickle.

$ sudo apt-get install -f
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  linux-headers-3.2.0-29 linux-headers-3.2.0-29-generic
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  linux-headers-3.2.0-80
The following NEW packages will be installed
  linux-headers-3.2.0-80
0 to upgrade, 1 to newly install, 0 to remove and 251 not to upgrade.
3 not fully installed or removed.
Need to get 0 B/11.7 MB of archives.
After this operation, 56.4 MB of additional disk space will be used.
Do you want to continue [Y/n]? 
(Reading database ... 1255202 files and directories currently installed.)
Unpacking linux-headers-3.2.0-80 (from .../linux-headers-3.2.0-80_3.2.0-80.116_all.deb) ...
dpkg: error processing /var/cache/apt/archives/linux-headers-3.2.0-80_3.2.0-80.116_all.deb (--unpack):
 unable to create `/usr/src/linux-headers-3.2.0-80/include/linux/sunrpc/gss_err.h.dpkg-new' (while processing `./usr/src/linux-headers-3.2.0-80/include/linux/sunrpc/gss_err.h'): No space left on device
No apport report written because the error message indicates a disk full error
                                                                              dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
 /var/cache/apt/archives/linux-headers-3.2.0-80_3.2.0-80.116_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

But:

us@desktop:/var/log$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda1       20158332 16166516   2967816  85% /
udev             4026512        4   4026508   1% /dev
tmpfs             807064      996    806068   1% /run
none                5120        0      5120   0% /run/lock
none             4035308      128   4035180   1% /run/shm
/dev/sda4      904990760 51533960 807485908   6% /home

Stumped!

us@desktop:~$ df -i
Filesystem       Inodes   IUsed    IFree IUse% Mounted on
/dev/sda1       1281120 1278499     2621  100% /
udev            1006628     507  1006121    1% /dev
tmpfs           1008827     445  1008382    1% /run
none            1008827       3  1008824    1% /run/lock
none            1008827       6  1008821    1% /run/shm
/dev/sda4      57466880  110192 57356688    1% /home

Oh so the inode table is full. Never had that before, what causes that then?

Answer 1

As shown in the output of df -i, you are running very low on number of free inodes on / and hence the dpkg fails to complete the operation.

Most of the time, this is caused by numerous small files created somewhere (unintentionally) in the filesystem.

You can check the files of size less than 1 KB and start removing them if not needed:

sudo find / -type f -size -1k -ls

Similarly you can check with increasing file sizes until you get to the root of the problem.

Answer 2

All credit to @heemayl for the pointers that resulted in this answer. I've accepted that answer, but here's the details of how I fixed it, in case it's useful to others.

The issue was caused by a full inode table. As I understand, there are two parts to a filesystem, inodes and space. inodes are related to the number of files, where as space is just how big they are. My system was chock-full of millions of tiny files caused by Ubuntu not removing old kernels. As the system has run for a long time I had over 40 kernels, incl. header files, installed which is too much for a 20Gb system partition's file system to take.

This meant dpkg could not complete an installation - it needed more space to complete its work - but unfortunately that meant neither could I create space by using apt to uninstall things!

The solution I found was to move all the /usr/src/ files to my other partition, leaving just a symlink to the files on the full fs:

sudo mv /usr/src /home/usr-src
sudo ln -s /home/usr-src /usr/src

After doing this I was able to let apt complete:

sudo apt install -f

And after that I was able to uninstall all the older kernels that I did not need, freeing up space, and then I could move the src folder back.