puzzled by "no space" complaint from apt

Question

Background: My / partition was at 93%, but I had some unallocated space so I created a new home for /usr . After the move the space usage looks like:

>df -k
Filesystem     1K-blocks     Used Available Use% Mounted on
//dev/sda1       15747484  6822736   8124800  46% /
udev             1954244        4   1954240   1% /dev
tmpfs             786052     1040    785012   1% /run
none                5120        0      5120   0% /run/lock
none             1965124      512   1964612   1% /run/shm
cgroup           1965124        0   1965124   0% /sys/fs/cgroup
/dev/sda5       62993332 30014168  29779216  51% /home
/dev/sda7       26540640       32  26540608   1% /windows
/dev/sda8       10320184  7156516   2639432  74% /usr

Now the system runs but when I allowed a normal software update, the kernel headers portion produced an error:

dpkg: error processing /var/cache/apt/archives/linux-headers-3.2.0-55_3.2.0-55.85_all.deb (--unpack):
 unable to create `/usr/src/linux-headers-3.2.0-55/arch/arm/mach-pxa/include/mach/eseries-irq.h.dpkg-new' (while processing `./usr/src/linux-headers-3.2.0-55/arch/arm/mach-pxa/include/mach/eseries-irq.h'): No space left on device

This "space" complaint and the df output above seem to contradict one another.

I get the same error when I try fixing the repository (sudo apt-get -f install). Thinking I had too many old kernel versions I tried removing some (sudo apt-get remove linux-headers-3.2.0-38 linux-headers-3.2.0-38-generic linux-image-3.2.0-36-generic ... ) but this produces an error and a suggestion that I try apt-get -f install .

Some system information:

> uname -a
Linux <hostname> 3.2.0-55-generic-pae #85-Ubuntu SMP Wed Oct 2 14:03:15 UTC 2013 i686 i686 i386 GNU/Linux
> cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.3 LTS"

I'm afraid I won't be able to update any software until I get this headers issue resolved. I hope someone out there can help.

Answer 1

The inodes have it. Thanks again, gertvdijk.

In my comment you can see that the filesystem, though not nearly full, was about out of inodes. After looking at man mkfs.ext4 I decided to cut its bytes/inodes ratio in half. This ratio is set by the -i option. A quick calculation based on the outputs of df -k and df -i showed mine to be about 2^14, so I would go with 2^13. (Below I've used {} to indicate placeholder objects whose details might be distracting.) I then

1. decided to use temporary storage {tempmountpoint} mounted on {tempdev}. This is big enough to hold the contents of my /usr tree.

2. copied the contents of /usr into {tempmountpoint} with

   rsync -avu /usr/ {tempmountpoint}
   

3. unmounted both of these

4. mounted /usr to {tempdev} (so I could keep working on other things through all of this)

5. reformatted /usr's original (and eventual target) partition with

   mkfs.ext4 -i 8192  /dev/sda8
   

6. mounted /dev/sda8 to {tempmountpoint}.

7. copied the contents of the temporary /usr back into its target partition with

   rsync -avu /usr/ {tempmountpoint} .
   

8. Finally, I looked up the (new) UUID with

   ls -l /dev/disk/by-uuid
   

9. used this UUID to edit the /usr entry in /etc/fstab, and

10. rebooted.

Updates are working normally now. And the filesystem looks to be in reasonable shape, as in:

> df -k /usr
Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/sda8       10156344 7349228   2282880  77% /usr
> df -i /usr
Filesystem      Inodes  IUsed  IFree IUse% Mounted on
/dev/sda8      1310720 679355 631365   52% /usr

Some notes:

(i) /home already had its own partition. If it had been in /usr I would approach this a little differently.

(ii) Choice of tools: I like ls /dev/disk/by-uuid because ls is very, very well known. I used rsync partly because when I had to do something else for a while before getting back to this, I could run the same rsync command again to "refresh" the copy. Also, I like it and it can do this. Others may have different utilities that satisfy these criteria. For example, until a few years ago I would've used cpio. Whatever gets you through the byte.

(iii) The placeholders refer to what had been the fat32 (vfat) filesystem on /dev/sda7, mounted at /windows. I reformatted it in cold blood before the above procedure. I never bothered changing the mountpoint name, which could be confusing second-hand.

Thanks to all who helped or contributed.

Answer 2

It says that you have no space left on /usr partition, you can try to remove something there.

Did you tried apt-get -f install?