3

I have a new Ubuntu Digital Ocean droplet which keeps running out of space on root.

df

Filesystem     1K-blocks      Used Available Use% Mounted on
udev             2013876         0   2013876   0% /dev
tmpfs             404640     10740    393900   3% /run
/dev/vda1       81120924  81104144       396 100% /          <----------<<<<< 396k space left after running apt clean commands. it keeps going down to 0 
tmpfs            2023196         0   2023196   0% /dev/shm
tmpfs               5120         0      5120   0% /run/lock
tmpfs            2023196         0   2023196   0% /sys/fs/cgroup
/dev/sda       363979364 245756248  99856652  72% /mnt/volume-nyc1-04
/dev/vda15        106858      3437    103421   4% /boot/efi
tmpfs             404640         0    404640   0% /run/user/1000
tmpfs             404640         0    404640   0% /run/user/0

the 80gig / partition is at 100%

but when i switch to / and do

du -sk -x -h *

16M bin
65M boot
0   dev
du: cannot access 'etc/ssl/certs/Camerfirma_Global_Chambersign_Root.pem': Structure needs cleaning
du: cannot access 'etc/ssl/certs/Certigna.pem': Structure needs cleaning
5.6M    etc
40M home
0   initrd.img
0   initrd.img.old
159M    lib
4.0K    lib64
16K lost+found
4.0K    media
8.0K    mnt
4.0K    opt
du: cannot access 'proc/4713/task/4713/fd/4': No such file or directory
du: cannot access 'proc/4713/task/4713/fdinfo/4': No such file or directory
du: cannot access 'proc/4713/fd/4': No such file or directory
du: cannot access 'proc/4713/fdinfo/4': No such file or directory
0   proc
32K root
11M run
14M sbin
8.0K    snap
4.0K    srv
0   sys
36K tmp
1.1G    usr
254M    var
0   vmlinuz
0   vmlinuz.old

i can't get it to add up past a few gigs. not sure where the whole 80 gigs go!

i tried a few apt clean commands so far but it hardly made a difference.

here is a list of bigger packages, but i am not sure which are safe to purge

dpkg-query --show --showformat='${Package;-50}\t${Installed-Size}\n' | sort -k 2 -n | grep -v deinstall | awk '{printf "%.3f MB \t %s\n", $2/(1024), $1}'

3.883 MB     ncurses-term
3.890 MB     libmagic1
3.958 MB     libdns162
4.039 MB     command-not-found-data
4.351 MB     linux-libc-dev
4.409 MB     libpython3.5
5.310 MB     geoip-database
6.102 MB     coreutils
6.307 MB     bitcoind
6.499 MB     dpkg
6.572 MB     udev
6.922 MB     perl-base
7.180 MB     linux-headers-4.4.0-133-generic
7.188 MB     linux-headers-4.4.0-134-generic
8.069 MB     lxd-client
8.517 MB     libpython2.7-stdlib
8.644 MB     python3.5-minimal
9.623 MB     libpython3.5-stdlib
10.696 MB    libc6
11.355 MB    grub-common
11.907 MB    libgcc-5-dev
12.921 MB    libc6-dev
13.023 MB    binutils
13.363 MB    locales
14.912 MB    libstdc++-5-dev
15.659 MB    iso-codes
16.714 MB    lxd
16.858 MB    perl-modules-5.22
18.414 MB    systemd
19.877 MB    libperl5.22
20.911 MB    cpp-5
22.343 MB    g++-5
23.000 MB    git
23.121 MB    gcc-5
26.240 MB    vim-runtime
29.389 MB    libicu55
38.940 MB    libpython2.7-dev
51.022 MB    libpython3.5-dev
64.915 MB    linux-image-4.4.0-131-generic
64.935 MB    linux-image-4.4.0-133-generic
64.947 MB    linux-image-4.4.0-134-generic
67.640 MB    linux-headers-4.4.0-133
67.645 MB    linux-headers-4.4.0-134
77.453 MB    snapd

when i do this.. ./mnt shows up as bigger than its size which is odd (drive is only 350 gigs). also what's up with all those access errors

sudo du -hd 1 | sort -h

du: cannot access './proc/5000/task/5000/fd/4': No such file or directory
du: cannot access './proc/5000/task/5000/fdinfo/4': No such file or directory
du: cannot access './proc/5000/fd/3': No such file or directory
du: cannot access './proc/5000/fdinfo/3': No such file or directory
du: cannot access './etc/ssl/certs/Camerfirma_Global_Chambersign_Root.pem': Structure needs cleaning
du: cannot access './etc/ssl/certs/Certigna.pem': Structure needs cleaning
0   ./dev
0   ./proc
0   ./sys
4.0K    ./lib64
4.0K    ./media
4.0K    ./opt
4.0K    ./srv
8.0K    ./snap
16K ./lost+found
32K ./root
36K ./tmp
5.6M    ./etc
11M ./run
14M ./sbin
16M ./bin
40M ./home
68M ./boot
159M    ./lib
254M    ./var
1.1G    ./usr
469G    ./mnt
471G    .

here are the contents of /mnt. could some of those be local files?

:/mnt# ls -al
total 20
drwxr-xr-x  5 root root 4096 Aug 27 18:04 .
drwxr-xr-x 23 root root 4096 Aug 27 16:33 ..
drwxr-xr-x  2 root root 4096 Aug 18 14:36 volume-nyc1-03
drwxr-xr-x  4 root root 4096 Aug 27 16:42 volume_nyc1_04
drwxr-xr-x  4 root root 4096 Aug 27 16:42 volume-nyc1-04


:/mnt# du -sk -x -h *
4.0K    volume-nyc1-03
235G    volume_nyc1_04

check disk is coming up with some stuff.

$:/# sudo fsck -nf /dev/vda1

fsck from util-linux 2.27.1
e2fsck 1.42.13 (17-May-2015)
Warning!  /dev/vda1 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Entry 'Camerfirma_Global_Chambersign_Root.pem' in /etc/ssl/certs (1490) has deleted/unused inode 1521.  Clear? no

Entry 'Camerfirma_Global_Chambersign_Root.pem' in /etc/ssl/certs (1490) has an incorrect filetype (was 7, should be 0).
Fix? no

Entry 'Certigna.pem' in /etc/ssl/certs (1490) has deleted/unused inode 1522.  Clear? no

Entry 'Certigna.pem' in /etc/ssl/certs (1490) has an incorrect filetype (was 7, should be 0).
Fix? no

Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences:  -16499
Fix? no

Inode bitmap differences:  -(1521--1522)
Fix? no


cloudimg-rootfs: ********** WARNING: Filesystem still has errors **********

cloudimg-rootfs: 109495/10321920 files (0.6% non-contiguous), 20938940/20943099 blocks
1

First, you can add the -x option to prevent du to traverse filesystems.

Then add sort -h to sort by filesize

Finally, run this command as root.

du -shx /* | sort -rh

This can help you figure out what's going on by looking at the first lines and without spending too much time exploring your /mnt.


You also seem to have filesystem corruption on /etc/ files. /etc/ files are unlikely to be big, but you can still check your filesystem for errors before it get worse.

If you can't figure out the space consumption, run the same command in /etc/, so you'll know the size it takes without the corrupted directory.

  • thanks that's helpful. but it still shows the same items just different order 1.1G /usr 254M /var 159M /lib 65M /boot. im aware /mnt is a different partition so not looking at it – Sonic Soul Aug 28 '18 at 15:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.