No space left on device even though there is

Question

A laptop with broken screen that I access via SSH. The sudo apt-get autoremove command gives me the following errors

cp: error writing ‘/tmp/mkinitramfs_zN6ZvT//lib/x86_64-linux-gnu/libpthread.so.0’: No space left on device
cp: failed to extend ‘/tmp/mkinitramfs_zN6ZvT//lib/x86_64-linux-gnu/libpthread.so.0’: No space left on device
cp: error writing ‘/tmp/mkinitramfs_zN6ZvT//sbin/modprobe’: No space left on device
cp: failed to extend ‘/tmp/mkinitramfs_zN6ZvT//sbin/modprobe’: No space left on device
cp: error writing ‘/tmp/mkinitramfs_zN6ZvT//sbin/rmmod’: No space left on device
cp: failed to extend ‘/tmp/mkinitramfs_zN6ZvT//sbin/rmmod’: No space left on device

I get similar errors from various other commands but gparted tells me there is more than 20GB space left on the (single) partition on the laptop. Here is the output of df

$ df -i
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
udev            502644    484  502160    1% /dev
tmpfs           505433    503  504930    1% /run
/dev/sda1      7331840 214087 7117753    3% /
none            505433      2  505431    1% /sys/fs/cgroup
none            505433      3  505430    1% /run/lock
none            505433      4  505429    1% /run/shm
none            505433     16  505417    1% /run/user
overflow        505433    401  505032    1% /tmp

And:

$ df -k
Filesystem     1K-blocks     Used Available Use% Mounted on
udev             2010576       12   2010564   1% /dev
tmpfs             404348     1284    403064   1% /run
/dev/sda1      115247656 83994028  25809372  77% /
none                   4        0         4   0% /sys/fs/cgroup
none                5120        0      5120   0% /run/lock
none             2021732      204   2021528   1% /run/shm
none              102400       16    102384   1% /run/user
overflow            1024     1024         0 100% /tmp

Error started after I ran sudo apt-get upgrade.

Answer 1

At some point in the past, your root filesystem filled up, and a small, temporary /tmp was created to allow boot to succeed. This small /tmp was never deleted, so now, even though you have room on /, you still are filling up the small /tmp and seeing your problem. Simply unmount it:

sudo umount /tmp

and of course, try to ensure your / is as clean as possible.

---

Normally, /tmp is just a part of the root (/) filesystem, no separate mount is needed, unless there are special circumstances, like running out of root filespace (when some daemon creates the one you see), or maybe you have / on a very slow media (like an USB flash stick) and want /tmp in ram for performance, even with limited space.

Answer 2

If you run into this problem, where you get errors that seem to indicate that the disk is full when it's not, make sure to also check the inode utilization.

You can use df -i to get a quick report on the used/available inodes for each mount point.

If you see that you are running very low, or out of, inodes then the next step is to identify which folder is holding up most inodes. Since each file and directory uses an inode, you could have a folder with hundreds of thousands of tiny, or empty files that are using up all the inodes. Usual suspects include: temp directory, website cache directories, package cache directories etc.

Use this command to get an ordered list of the subdirectories with the most inodes used:

sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

Run this in your root folder, then drill down until you find your culprit.

Answer 3

I believe you have a lot of unused files remove them with:

sudo apt autoremove

Then re-check your space with df command

Answer 4

Your /tmp directory is set to overflow, so there is not enough disk space in that directory to perform apt-get operations

you can for your terminal session change the location of tmp for apt-get in order to perform the operation

mkdir -p /home/<user>/tmp
export TMPDIR=/home/<user>/tmp

Answer 5

To see how many inodes per folder:

du * -s --inodes