0

Im deploying to my server, using ansible, part of which runs a bunch of "update" commands before I start installing/updating packages. I recently got an error about disk space, but appear to be going in circles trying to fix it, below is as much informationas I can think to provide as to what has happened.

starting error:

sudo apt-get --fix-broken install -y

Gave the following:

sudo: unable to resolve host myserver-001
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following extra packages will be installed:
linux-headers-3.13.0-160
The following NEW packages will be installed:
linux-headers-3.13.0-160
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
4 not fully installed or removed.
Need to get 0 B/8,911 kB of archives.
After this operation, 63.6 MB of additional disk space will be used.
(Reading database ... 505365 files and directories currently installed.)
Preparing to unpack .../linux-headers-3.13.0-160_3.13.0-160.210_all.deb ...
Unpacking linux-headers-3.13.0-160 (3.13.0-160.210) ...
dpkg: error processing archive /var/cache/apt/archives/linux-headers-3.13.0-160_3.13.0-160.210_all.deb (--unpack):
error creating directory `./usr/src/linux-headers-3.13.0-160/drivers/staging/lustre/lustre/osc': No space left on device
No apport report written because the error message indicates a disk full error
dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
/var/cache/apt/archives/linux-headers-3.13.0-160_3.13.0-160.210_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

For me, the important part was:

error creating directory `./usr/src/linux-headers-3.13.0-160/drivers/staging/lustre/lustre/osc': No space left on device  

No apport report written because the error message indicates a disk full error

So, after some googling, I ended up at these places: /boot and /usr/src/, and checking innodes

df /boot | df /usr/src:

Filesystem     1K-blocks    Used Available Use% Mounted on
/dev/xvda1       8115168 5474876   2205016  72% /

df -i:

Filesystem     Inodes  IUsed  IFree IUse% Mounted on
udev           480531    391 480140    1% /dev
tmpfs          481833    312 481521    1% /run
/dev/xvda1     524288 521345   2943  100% /
none           481833      2 481831    1% /sys/fs/cgroup
none           481833      1 481832    1% /run/lock
none           481833      1 481832    1% /run/shm
none           481833      2 481831    1% /run/user
/dev/xvdb      262144     11 262133    1% /mnt

uname -r:

3.13.0-158-generic

I'm not great at this stuff, hence trying to provide as much information as possible. I apologise if any of this is useless.

In the /usr/src I have a lot of linux-headers:

drwxr-xr-x 24 root root 4096 Nov 24 2016 linux-headers-3.13.0-101
drwxr-xr-x 7 root root 4096 Nov 24 2016 linux-headers-3.13.0-101-generic
drwxr-xr-x 24 root root 4096 Jan 20 2017 linux-headers-3.13.0-107
drwxr-xr-x 7 root root 4096 Jan 20 2017 linux-headers-3.13.0-107-generic
drwxr-xr-x 24 root root 4096 Mar 9 2017 linux-headers-3.13.0-112
drwxr-xr-x 7 root root 4096 Mar 9 2017 linux-headers-3.13.0-112-generic
drwxr-xr-x 24 root root 4096 Apr 7 2017 linux-headers-3.13.0-116
drwxr-xr-x 7 root root 4096 Apr 7 2017 linux-headers-3.13.0-116-generic
drwxr-xr-x 24 root root 4096 May 4 2017 linux-headers-3.13.0-117
drwxr-xr-x 7 root root 4096 May 4 2017 linux-headers-3.13.0-117-generic
drwxr-xr-x 24 root root 4096 May 18 2017 linux-headers-3.13.0-119
drwxr-xr-x 7 root root 4096 May 18 2017 linux-headers-3.13.0-119-generic
drwxr-xr-x 24 root root 4096 Jun 22 2017 linux-headers-3.13.0-121
drwxr-xr-x 7 root root 4096 Jun 22 2017 linux-headers-3.13.0-121-generic
drwxr-xr-x 24 root root 4096 Jul 7 2017 linux-headers-3.13.0-123
drwxr-xr-x 7 root root 4096 Jul 7 2017 linux-headers-3.13.0-123-generic
drwxr-xr-x 24 root root 4096 Jul 18 2017 linux-headers-3.13.0-125
drwxr-xr-x 7 root root 4096 Jul 18 2017 linux-headers-3.13.0-125-generic
drwxr-xr-x 24 root root 4096 Aug 31 2017 linux-headers-3.13.0-129
drwxr-xr-x 7 root root 4096 Aug 31 2017 linux-headers-3.13.0-129-generic
drwxr-xr-x 24 root root 4096 Jan 25 2018 linux-headers-3.13.0-141
drwxr-xr-x 7 root root 4096 Jan 25 2018 linux-headers-3.13.0-141-generic
drwxr-xr-x 24 root root 4096 Feb 22 2018 linux-headers-3.13.0-142
drwxr-xr-x 7 root root 4096 Feb 22 2018 linux-headers-3.13.0-142-generic
drwxr-xr-x 24 root root 4096 Mar 27 2018 linux-headers-3.13.0-143
drwxr-xr-x 7 root root 4096 Mar 27 2018 linux-headers-3.13.0-143-generic
drwxr-xr-x 24 root root 4096 Apr 27 13:47 linux-headers-3.13.0-145
drwxr-xr-x 7 root root 4096 Apr 27 13:47 linux-headers-3.13.0-145-generic
drwxr-xr-x 24 root root 4096 May 14 11:11 linux-headers-3.13.0-147
drwxr-xr-x 7 root root 4096 May 14 11:11 linux-headers-3.13.0-147-generic
drwxr-xr-x 24 root root 4096 May 23 11:04 linux-headers-3.13.0-149
drwxr-xr-x 7 root root 4096 May 23 11:04 linux-headers-3.13.0-149-generic
drwxr-xr-x 24 root root 4096 Sep 12 15:19 linux-headers-3.13.0-158
drwxr-xr-x 7 root root 4096 Sep 12 15:19 linux-headers-3.13.0-158-generic
drwxr-xr-x 7 root root 4096 Oct 12 11:15 linux-headers-3.13.0-160-generic
drwxr-xr-x 24 root root 4096 Mar 25 2015 linux-headers-3.13.0-48
drwxr-xr-x 7 root root 4096 Mar 25 2015 linux-headers-3.13.0-48-generic

I assume I don't need many of these, but when I try to to uninstall the 2015 kernels, I get the same error as before, which is complaining about the most recent and the fact it can not install its dependency:

sudo apt-get remove linux-headers-3.13.0-48

sudo: unable to resolve host myserver-001
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
linux-headers-3.13.0-160-generic : Depends: linux-headers-3.13.0-160 but it is not going to be installed
linux-headers-3.13.0-48-generic : Depends: linux-headers-3.13.0-48 but it is not going to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).

If I try "apt-get -f install", I get same message as before about no space.

/boot ls

 abi-3.13.0-101-generic     config-3.13.0-149-generic      System.map-3.13.0-117-generic
 abi-3.13.0-107-generic     config-3.13.0-158-generic      System.map-3.13.0-119-generic
 abi-3.13.0-112-generic     config-3.13.0-160-generic      System.map-3.13.0-121-generic
 abi-3.13.0-116-generic     config-3.13.0-48-generic       System.map-3.13.0-123-generic
 abi-3.13.0-117-generic     grub                           System.map-3.13.0-125-generic
 abi-3.13.0-119-generic     initrd.img-3.13.0-101-generic  System.map-3.13.0-129-generic
 abi-3.13.0-121-generic     initrd.img-3.13.0-107-generic  System.map-3.13.0-141-generic
 abi-3.13.0-123-generic     initrd.img-3.13.0-112-generic  System.map-3.13.0-142-generic
 abi-3.13.0-125-generic     initrd.img-3.13.0-116-generic  System.map-3.13.0-143-generic
 abi-3.13.0-129-generic     initrd.img-3.13.0-117-generic  System.map-3.13.0-145-generic
 abi-3.13.0-141-generic     initrd.img-3.13.0-119-generic  System.map-3.13.0-147-generic
 abi-3.13.0-142-generic     initrd.img-3.13.0-121-generic  System.map-3.13.0-149-generic
 abi-3.13.0-143-generic     initrd.img-3.13.0-123-generic  System.map-3.13.0-158-generic
 abi-3.13.0-145-generic     initrd.img-3.13.0-125-generic  System.map-3.13.0-160-generic
 abi-3.13.0-147-generic     initrd.img-3.13.0-129-generic  System.map-3.13.0-48-generic
 abi-3.13.0-149-generic     initrd.img-3.13.0-141-generic  vmlinuz-3.13.0-101-generic
 abi-3.13.0-158-generic     initrd.img-3.13.0-142-generic  vmlinuz-3.13.0-107-generic
 abi-3.13.0-160-generic     initrd.img-3.13.0-143-generic  vmlinuz-3.13.0-112-generic
 abi-3.13.0-48-generic      initrd.img-3.13.0-145-generic  vmlinuz-3.13.0-116-generic
 config-3.13.0-101-generic  initrd.img-3.13.0-147-generic  vmlinuz-3.13.0-117-generic
 config-3.13.0-107-generic  initrd.img-3.13.0-149-generic  vmlinuz-3.13.0-119-generic
 config-3.13.0-112-generic  initrd.img-3.13.0-158-generic  vmlinuz-3.13.0-121-generic
 config-3.13.0-116-generic  initrd.img-3.13.0-160-generic  vmlinuz-3.13.0-123-generic
 config-3.13.0-117-generic  initrd.img-3.13.0-48-generic   vmlinuz-3.13.0-125-generic
 config-3.13.0-119-generic  retpoline-3.13.0-143-generic   vmlinuz-3.13.0-129-generic
 config-3.13.0-121-generic  retpoline-3.13.0-145-generic   vmlinuz-3.13.0-141-generic
 config-3.13.0-123-generic  retpoline-3.13.0-147-generic   vmlinuz-3.13.0-142-generic
 config-3.13.0-125-generic  retpoline-3.13.0-149-generic   vmlinuz-3.13.0-143-generic
 config-3.13.0-129-generic  retpoline-3.13.0-158-generic   vmlinuz-3.13.0-145-generic
 config-3.13.0-141-generic  retpoline-3.13.0-160-generic   vmlinuz-3.13.0-147-generic
 config-3.13.0-142-generic  System.map-3.13.0-101-generic  vmlinuz-3.13.0-149-generic
 config-3.13.0-143-generic  System.map-3.13.0-107-generic  vmlinuz-3.13.0-158-generic
 config-3.13.0-145-generic  System.map-3.13.0-112-generic  vmlinuz-3.13.0-160-generic
 config-3.13.0-147-generic  System.map-3.13.0-116-generic  vmlinuz-3.13.0-48-generic

I tried removing the latest kernel from here - that seems to have been partially installed: sudo apt-get purge linux-image-3.13.0-160

Gave me this:

Reading package lists... Done
Building dependency tree
Reading state information... Done
Note, selecting 'linux-image-3.13.0-160-lowlatency' for regex 'linux-image-3.13.0-160'
Note, selecting 'linux-image-3.13.0-160-generic' for regex 'linux-image-3.13.0-160'
Package 'linux-image-3.13.0-160-lowlatency' is not installed, so not removed
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies:
linux-headers-3.13.0-160-generic : Depends: linux-headers-3.13.0-160 but it is not going to be installed
linux-image-virtual : Depends: linux-image-3.13.0-160-generic but it is not going to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).

I'm really confused as to what to do. If anyone can point me in the right direction, that would be great! Am I go to just use rm to remove some of the old files in either /boot or /usr/src ? Cos most posts suggested not to do this, but to use the apt-get remove etc.. commands Cheers

  • Welcome to Ask Ubuntu!!, you have sudo: unable to resolve host myserver-001 to fixed it, try to type sudo nano /etc/hosts and change 127.0.1.1 myserver-001 saved and try sudo apt-get --fix missing && sudo apt-get update. – abu-ahmed al-khatiri Oct 15 '18 at 10:14
  • I didn't read it all sorry, but I stopped when I saw inodes 100% used... Inodes are blocks/clusters, and my guess is you have some program that's created tons of little files (eg. email logs of events) which have used all your disk space (inodes anyway). Yes you can 'grow' files, but can't create new files (due lack of inodes). I'd explore your filesystem using du -hs --inodes for a directory with tons of little files that has used all inodes (& clean/fix it). Inodes are finite & created at format/mkfs... My 2c (this possibly has created your apt issues too which needs fix too) – guiverc Oct 15 '18 at 11:17
  • @abu-ahmedal-khatiri ive fixed the "unable to resolve host" but that wasnt causing this issue. Your next commands gave me same error as above. (linux-headers.*.160 depends on ... unmet dependencies), following this up by passing -f, gives me the first error of "no space" – datacubed Oct 15 '18 at 13:40
  • @guiverc that is what i think it is, but i'm unsure how to go about this. Your command returns "unrecognised options '--inodes'" – datacubed Oct 15 '18 at 13:41
  • 3
0

You are out indeed of inodes (you knew that already).

The condition is indeed caused by all those kernel header packages (you knew that, too).

Apt cannot do the uninstalls because completing the in-process installs are earlier in the queue...and those in-process installs abort due to out-of-inodes. Apt always tries to complete in-process actions before new actions. (you figured that out, too)

The correct easy trick here is to uninstall the first kernel header package using dpkg instead of apt. This will free enough inodes to unclog apt. Subsequent uninstalls can be done using apt.

sudo dpkg --remove linux-headers-3.13.0-48 linux-headers-3.13.0-48-generic
sudo apt-get remove linux-headers-3.13.0-101 linux-headers-3.13.0-101-generic

There's a bit more cleaning up to be done - lots more kernel headers, plus lots of obsolete kernel images hanging out in /boot.

Pro Tip: NEVER use rm to remove files placed by the package manager. You will simply replace today's out-of-space error with next week's missing-file error...which can be much harder to fix. ALWAYS use the package manager to remove files that it placed.

  • This worked. Thank you so much. How do I stay on top of the kernel headers? I have a few servers to deploy to, and would rather not have to manually dial in to each one and run this command. IF they all have the same issue. I'm going to deploy to the 2nd box once this has finished. If that has the same issue, is there a command I can add to my ansible deploy script to easily clean this up old kernels? Also, is this worth me doing on this box1? – datacubed Oct 15 '18 at 15:15
  • Staying on top of kernel headers depends upon why they are installed in the first place - what's pulling them in (headers are not installed by default). Start with apt-get autoremove and see if that uninstalls them. Autoremove works when two conditions are true: apt-mark is 'auto' and nothing apt-marked 'manual' depends (directly or indirectly) upon it. If autoremove doesn't work, you must be a detective to figure out which condition is false and why. – user535733 Oct 15 '18 at 16:06
0

Try with installing Ubuntu tweaks. It will at least help you remove the old kernels and you can claim some free space.

Also try following commands:

sudo apt-get install --fix-missing --fix-broken

sudo dpkg --configure -a

Or Use a specific package name with error to reconfigure it

sudo dpkg-reconfigure <pakcage name>

[Edit]

Since you do not have space to do anything, it is better to truncate a few error /access log files.

Use truncate --size 0 [path to the log file]

If you want to retain the content of those files, first download them on your local machine. In your DB, if the system is using any type of logging/caching, empty those tables.

  • The first command, gives me the same error as above about the linux-headers.*.160 kernel file and that it needs space to install its dependency – datacubed Oct 15 '18 at 13:36
  • That is where you can use the Ubuntu tweaks – Bhavin Joshi Oct 15 '18 at 13:41
  • Sounds funny, right? But a tiny utility won't heart. – Bhavin Joshi Oct 15 '18 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.