Trying to free up space for some updates, but havent dependency issues trying to remove/purge

Question

Im deploying to my server, using ansible, part of which runs a bunch of "update" commands before I start installing/updating packages. I recently got an error about disk space, but appear to be going in circles trying to fix it, below is as much informationas I can think to provide as to what has happened.

starting error:

sudo apt-get --fix-broken install -y

Gave the following:

sudo: unable to resolve host myserver-001
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following extra packages will be installed:
linux-headers-3.13.0-160
The following NEW packages will be installed:
linux-headers-3.13.0-160
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
4 not fully installed or removed.
Need to get 0 B/8,911 kB of archives.
After this operation, 63.6 MB of additional disk space will be used.
(Reading database ... 505365 files and directories currently installed.)
Preparing to unpack .../linux-headers-3.13.0-160_3.13.0-160.210_all.deb ...
Unpacking linux-headers-3.13.0-160 (3.13.0-160.210) ...
dpkg: error processing archive /var/cache/apt/archives/linux-headers-3.13.0-160_3.13.0-160.210_all.deb (--unpack):
error creating directory `./usr/src/linux-headers-3.13.0-160/drivers/staging/lustre/lustre/osc': No space left on device
No apport report written because the error message indicates a disk full error
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
/var/cache/apt/archives/linux-headers-3.13.0-160_3.13.0-160.210_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

For me, the important part was:

error creating directory `./usr/src/linux-headers-3.13.0-160/drivers/staging/lustre/lustre/osc': No space left on device  

No apport report written because the error message indicates a disk full error

So, after some googling, I ended up at these places: /boot and /usr/src/, and checking innodes

df /boot | df /usr/src:

Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1       8115168 5474876   2205016  72% /

df -i:

Filesystem     Inodes  IUsed  IFree IUse% Mounted on
udev           480531    391 480140    1% /dev
tmpfs          481833    312 481521    1% /run
/dev/xvda1     524288 521345   2943  100% /
none           481833      2 481831    1% /sys/fs/cgroup
none           481833      1 481832    1% /run/lock
none           481833      1 481832    1% /run/shm
none           481833      2 481831    1% /run/user
/dev/xvdb      262144     11 262133    1% /mnt

uname -r:

3.13.0-158-generic

I'm not great at this stuff, hence trying to provide as much information as possible. I apologise if any of this is useless.

In the /usr/src I have a lot of linux-headers:

drwxr-xr-x 24 root root 4096 Nov 24 2016 linux-headers-3.13.0-101
drwxr-xr-x 7 root root 4096 Nov 24 2016 linux-headers-3.13.0-101-generic
drwxr-xr-x 24 root root 4096 Jan 20 2017 linux-headers-3.13.0-107
drwxr-xr-x 7 root root 4096 Jan 20 2017 linux-headers-3.13.0-107-generic
drwxr-xr-x 24 root root 4096 Mar 9 2017 linux-headers-3.13.0-112
drwxr-xr-x 7 root root 4096 Mar 9 2017 linux-headers-3.13.0-112-generic
drwxr-xr-x 24 root root 4096 Apr 7 2017 linux-headers-3.13.0-116
drwxr-xr-x 7 root root 4096 Apr 7 2017 linux-headers-3.13.0-116-generic
drwxr-xr-x 24 root root 4096 May 4 2017 linux-headers-3.13.0-117
drwxr-xr-x 7 root root 4096 May 4 2017 linux-headers-3.13.0-117-generic
drwxr-xr-x 24 root root 4096 May 18 2017 linux-headers-3.13.0-119
drwxr-xr-x 7 root root 4096 May 18 2017 linux-headers-3.13.0-119-generic
drwxr-xr-x 24 root root 4096 Jun 22 2017 linux-headers-3.13.0-121
drwxr-xr-x 7 root root 4096 Jun 22 2017 linux-headers-3.13.0-121-generic
drwxr-xr-x 24 root root 4096 Jul 7 2017 linux-headers-3.13.0-123
drwxr-xr-x 7 root root 4096 Jul 7 2017 linux-headers-3.13.0-123-generic
drwxr-xr-x 24 root root 4096 Jul 18 2017 linux-headers-3.13.0-125
drwxr-xr-x 7 root root 4096 Jul 18 2017 linux-headers-3.13.0-125-generic
drwxr-xr-x 24 root root 4096 Aug 31 2017 linux-headers-3.13.0-129
drwxr-xr-x 7 root root 4096 Aug 31 2017 linux-headers-3.13.0-129-generic
drwxr-xr-x 24 root root 4096 Jan 25 2018 linux-headers-3.13.0-141
drwxr-xr-x 7 root root 4096 Jan 25 2018 linux-headers-3.13.0-141-generic
drwxr-xr-x 24 root root 4096 Feb 22 2018 linux-headers-3.13.0-142
drwxr-xr-x 7 root root 4096 Feb 22 2018 linux-headers-3.13.0-142-generic
drwxr-xr-x 24 root root 4096 Mar 27 2018 linux-headers-3.13.0-143
drwxr-xr-x 7 root root 4096 Mar 27 2018 linux-headers-3.13.0-143-generic
drwxr-xr-x 24 root root 4096 Apr 27 13:47 linux-headers-3.13.0-145
drwxr-xr-x 7 root root 4096 Apr 27 13:47 linux-headers-3.13.0-145-generic
drwxr-xr-x 24 root root 4096 May 14 11:11 linux-headers-3.13.0-147
drwxr-xr-x 7 root root 4096 May 14 11:11 linux-headers-3.13.0-147-generic
drwxr-xr-x 24 root root 4096 May 23 11:04 linux-headers-3.13.0-149
drwxr-xr-x 7 root root 4096 May 23 11:04 linux-headers-3.13.0-149-generic
drwxr-xr-x 24 root root 4096 Sep 12 15:19 linux-headers-3.13.0-158
drwxr-xr-x 7 root root 4096 Sep 12 15:19 linux-headers-3.13.0-158-generic
drwxr-xr-x 7 root root 4096 Oct 12 11:15 linux-headers-3.13.0-160-generic
drwxr-xr-x 24 root root 4096 Mar 25 2015 linux-headers-3.13.0-48
drwxr-xr-x 7 root root 4096 Mar 25 2015 linux-headers-3.13.0-48-generic

I assume I don't need many of these, but when I try to to uninstall the 2015 kernels, I get the same error as before, which is complaining about the most recent and the fact it can not install its dependency:

sudo apt-get remove linux-headers-3.13.0-48

sudo: unable to resolve host myserver-001
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
linux-headers-3.13.0-160-generic : Depends: linux-headers-3.13.0-160 but it is not going to be installed
linux-headers-3.13.0-48-generic : Depends: linux-headers-3.13.0-48 but it is not going to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).

If I try "apt-get -f install", I get same message as before about no space.

/boot ls

 abi-3.13.0-101-generic     config-3.13.0-149-generic      System.map-3.13.0-117-generic
 abi-3.13.0-107-generic     config-3.13.0-158-generic      System.map-3.13.0-119-generic
 abi-3.13.0-112-generic     config-3.13.0-160-generic      System.map-3.13.0-121-generic
 abi-3.13.0-116-generic     config-3.13.0-48-generic       System.map-3.13.0-123-generic
 abi-3.13.0-117-generic     grub                           System.map-3.13.0-125-generic
 abi-3.13.0-119-generic     initrd.img-3.13.0-101-generic  System.map-3.13.0-129-generic
 abi-3.13.0-121-generic     initrd.img-3.13.0-107-generic  System.map-3.13.0-141-generic
 abi-3.13.0-123-generic     initrd.img-3.13.0-112-generic  System.map-3.13.0-142-generic
 abi-3.13.0-125-generic     initrd.img-3.13.0-116-generic  System.map-3.13.0-143-generic
 abi-3.13.0-129-generic     initrd.img-3.13.0-117-generic  System.map-3.13.0-145-generic
 abi-3.13.0-141-generic     initrd.img-3.13.0-119-generic  System.map-3.13.0-147-generic
 abi-3.13.0-142-generic     initrd.img-3.13.0-121-generic  System.map-3.13.0-149-generic
 abi-3.13.0-143-generic     initrd.img-3.13.0-123-generic  System.map-3.13.0-158-generic
 abi-3.13.0-145-generic     initrd.img-3.13.0-125-generic  System.map-3.13.0-160-generic
 abi-3.13.0-147-generic     initrd.img-3.13.0-129-generic  System.map-3.13.0-48-generic
 abi-3.13.0-149-generic     initrd.img-3.13.0-141-generic  vmlinuz-3.13.0-101-generic
 abi-3.13.0-158-generic     initrd.img-3.13.0-142-generic  vmlinuz-3.13.0-107-generic
 abi-3.13.0-160-generic     initrd.img-3.13.0-143-generic  vmlinuz-3.13.0-112-generic
 abi-3.13.0-48-generic      initrd.img-3.13.0-145-generic  vmlinuz-3.13.0-116-generic
 config-3.13.0-101-generic  initrd.img-3.13.0-147-generic  vmlinuz-3.13.0-117-generic
 config-3.13.0-107-generic  initrd.img-3.13.0-149-generic  vmlinuz-3.13.0-119-generic
 config-3.13.0-112-generic  initrd.img-3.13.0-158-generic  vmlinuz-3.13.0-121-generic
 config-3.13.0-116-generic  initrd.img-3.13.0-160-generic  vmlinuz-3.13.0-123-generic
 config-3.13.0-117-generic  initrd.img-3.13.0-48-generic   vmlinuz-3.13.0-125-generic
 config-3.13.0-119-generic  retpoline-3.13.0-143-generic   vmlinuz-3.13.0-129-generic
 config-3.13.0-121-generic  retpoline-3.13.0-145-generic   vmlinuz-3.13.0-141-generic
 config-3.13.0-123-generic  retpoline-3.13.0-147-generic   vmlinuz-3.13.0-142-generic
 config-3.13.0-125-generic  retpoline-3.13.0-149-generic   vmlinuz-3.13.0-143-generic
 config-3.13.0-129-generic  retpoline-3.13.0-158-generic   vmlinuz-3.13.0-145-generic
 config-3.13.0-141-generic  retpoline-3.13.0-160-generic   vmlinuz-3.13.0-147-generic
 config-3.13.0-142-generic  System.map-3.13.0-101-generic  vmlinuz-3.13.0-149-generic
 config-3.13.0-143-generic  System.map-3.13.0-107-generic  vmlinuz-3.13.0-158-generic
 config-3.13.0-145-generic  System.map-3.13.0-112-generic  vmlinuz-3.13.0-160-generic
 config-3.13.0-147-generic  System.map-3.13.0-116-generic  vmlinuz-3.13.0-48-generic

I tried removing the latest kernel from here - that seems to have been partially installed: sudo apt-get purge linux-image-3.13.0-160

Gave me this:

Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'linux-image-3.13.0-160-lowlatency' for regex 'linux-image-3.13.0-160'
Note, selecting 'linux-image-3.13.0-160-generic' for regex 'linux-image-3.13.0-160'
Package 'linux-image-3.13.0-160-lowlatency' is not installed, so not removed
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
linux-headers-3.13.0-160-generic : Depends: linux-headers-3.13.0-160 but it is not going to be installed
linux-image-virtual : Depends: linux-image-3.13.0-160-generic but it is not going to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).

I'm really confused as to what to do. If anyone can point me in the right direction, that would be great! Am I go to just use rm to remove some of the old files in either /boot or /usr/src ? Cos most posts suggested not to do this, but to use the apt-get remove etc.. commands Cheers

Answer 1

You are out indeed of inodes (you knew that already).

The condition is indeed caused by all those kernel header packages (you knew that, too).

Apt cannot do the uninstalls because completing the in-process installs are earlier in the queue...and those in-process installs abort due to out-of-inodes. Apt always tries to complete in-process actions before new actions. (you figured that out, too)

The correct easy trick here is to uninstall the first kernel header package using dpkg instead of apt. This will free enough inodes to unclog apt. Subsequent uninstalls can be done using apt.

sudo dpkg --remove linux-headers-3.13.0-48 linux-headers-3.13.0-48-generic
sudo apt-get remove linux-headers-3.13.0-101 linux-headers-3.13.0-101-generic

There's a bit more cleaning up to be done - lots more kernel headers, plus lots of obsolete kernel images hanging out in /boot.

Pro Tip: NEVER use rm to remove files placed by the package manager. You will simply replace today's out-of-space error with next week's missing-file error...which can be much harder to fix. ALWAYS use the package manager to remove files that it placed.

Answer 2

Try with installing Ubuntu tweaks. It will at least help you remove the old kernels and you can claim some free space.

Also try following commands:

sudo apt-get install --fix-missing --fix-broken

sudo dpkg --configure -a

Or Use a specific package name with error to reconfigure it

sudo dpkg-reconfigure <pakcage name>

[Edit]

Since you do not have space to do anything, it is better to truncate a few error /access log files.

Use truncate --size 0 [path to the log file]

If you want to retain the content of those files, first download them on your local machine. In your DB, if the system is using any type of logging/caching, empty those tables.