1

I have created a new Ubuntu 20.04 VM for my docker host. I then moved my backup over to the new host and restarted all my containers. Apart from docker, docker-compose and ncdu nothing else has been installed on this host.

The VM has a 450GB disk and apparently 340GB are in use. I can not locate what is using all this diskspace though. All docker relevant mounts are in /home/docker/services/. NCDU reports 114.6 GiB disk usage which does not agree with df -h.

So my question is, what is happening here? What is causing this discrepancy?

docker@docker:~$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                                63G     0   63G   0% /dev
tmpfs                               13G  4.0M   13G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  440G  340G   78G  82% /
tmpfs                               63G     0   63G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                               63G     0   63G   0% /sys/fs/cgroup
/dev/sda2                          976M  104M  806M  12% /boot
/dev/sdc1                          157G   61M  149G   1% /elastic
/dev/loop0                          55M   55M     0 100% /snap/core18/1880
/dev/sdb1                          1.5T   81G  1.4T   6% /downloads
/dev/loop1                          56M   56M     0 100% /snap/core18/1944
/dev/loop2                          72M   72M     0 100% /snap/lxd/16099
/dev/loop3                          32M   32M     0 100% /snap/snapd/10492
/dev/loop4                          30M   30M     0 100% /snap/snapd/8542
/dev/loop5                          68M   68M     0 100% /snap/lxd/18150
//192.168.178.200/4k_series         79T   66T   13T  85% /4k_series
//192.168.178.200/4k_movies         79T   66T   13T  85% /4k_movies
//192.168.178.200/Docker_Backup     79T   66T   13T  85% /docker_backup
//192.168.178.200/Coding            79T   66T   13T  85% /coding
//192.168.178.200/Music             79T   66T   13T  85% /music
//192.168.178.200/eBooks            79T   66T   13T  85% /ebooks
//192.168.178.200/Cloud             79T   66T   13T  85% /cloud
//192.168.178.200/Movies            79T   66T   13T  84% /movies
//192.168.178.200/Audiobooks        79T   66T   13T  85% /audiobooks
//192.168.178.200/Fred              79T   66T   13T  85% /fred
//192.168.178.200/Series            79T   66T   13T  85% /series
//192.168.178.200/Photos            79T   66T   13T  85% /photos
tmpfs                               13G     0   13G   0% /run/user/1000

ncdu has been run as root output:

ncdu 1.14.1 ~ Use the arrow keys to navigate, press ? for help                                                                                                    
--- / ------------------------------------------------------------------------------------------------------------------------------------------------------------
   77.6 GiB [##########] /home                                                                                                                                    
   25.9 GiB [###       ] /var
    8.0 GiB [#         ]  swap.img
    2.1 GiB [          ] /usr
    1.0 GiB [          ] /snap
  101.0 MiB [          ] /boot
    5.3 MiB [          ] /etc
    3.9 MiB [          ] /run
   60.0 KiB [          ] /tmp
   52.0 KiB [          ] /root
   20.0 KiB [          ] /elastic
   16.0 KiB [          ] /opt
e  16.0 KiB [          ] /lost+found
e   4.0 KiB [          ] /srv
e   4.0 KiB [          ] /mnt
e   4.0 KiB [          ] /media
e   4.0 KiB [          ] /incomplete
e   4.0 KiB [          ] /cdrom
.   0.0   B [          ] /proc
    0.0   B [          ] /sys
    0.0   B [          ] /dev
@   0.0   B [          ]  libx32
@   0.0   B [          ]  lib64
@   0.0   B [          ]  lib32
@   0.0   B [          ]  sbin
@   0.0   B [          ]  lib
@   0.0   B [          ]  bin
<   0.0   B [          ]  series
<   0.0   B [          ]  photos
<   0.0   B [          ]  music
<   0.0   B [          ]  movies
<   0.0   B [          ]  fred
<   0.0   B [          ]  ebooks
<   0.0   B [          ]  downloads
<   0.0   B [          ]  docker_backup
<   0.0   B [          ]  coding
<   0.0   B [          ]  cloud
<   0.0   B [          ]  audiobooks
<   0.0   B [          ]  4k_series
<   0.0   B [          ]  4k_movies

 Total disk usage: 114.6 GiB  Apparent size: 128.1 TiB  Items: 4296902  

My fstab:

docker@docker:~$ cat /etc/fstab 
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/ubuntu-vg/ubuntu-lv during curtin installation
/dev/disk/by-id/dm-uuid-LVM-2qMn7z44yFIFNUpjG0awRAYnY86pTJbxy0DB3uCLpkalGZFRG023GYBeeazYfbe4 / ext4 defaults 0 0

# Server mounts
//192.168.178.200/Fred /fred cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Audiobooks /audiobooks cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Series /series cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Movies /movies cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/eBooks /ebooks cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Music /music cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Coding /coding cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Docker_Backup /docker_backup cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Cloud /cloud cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/Photos /photos cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/4k_series /4k_series cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0
//192.168.178.200/4k_movies /4k_movies cifs credentials=/home/docker/.smbcredentials,iocharset=utf8,gid=1000,uid=1000,file_mode=0777,dir_mode=0777 0 0


# /boot was on /dev/sda2 during curtin installation

/dev/disk/by-uuid/d47fc774-f9fe-40e8-a185-3a764ce6f971 /boot ext4 defaults 0 0
/swap.img       none    swap    sw      0       0

# secondary mounts
/dev/sdb1 /downloads ext4 defaults 0 0
/dev/sdc1 /elastic ext4 defaults 0 0  

Output of du -hs /

sudo du -hs / --exclude /movies --exclude /series --exclude /photos --exclude /4k_series --exclude /4k_movies --exclude /fred --exclude /music --exclude /docker_backup --exclude /coding --exclude /cloud --exclude /audiobooks --exclude /ebooks --exclude /downloads --exclude /elastic
[sudo] password for docker: 
du: cannot access '/proc/3471/task/355748/fd/12': No such file or directory
du: cannot access '/proc/5906/task/1546871/fd/449': No such file or directory
du: cannot access '/proc/1661328': No such file or directory
du: cannot access '/proc/1671145': No such file or directory
du: cannot access '/proc/1675436/task/1675436/fd/4': No such file or directory
du: cannot access '/proc/1675436/task/1675436/fdinfo/4': No such file or directory
du: cannot access '/proc/1675436/fd/3': No such file or directory
du: cannot access '/proc/1675436/fdinfo/3': No such file or directory
du: cannot access '/proc/1675471': No such file or directory
du: cannot access '/proc/1675472': No such file or directory
du: cannot access '/proc/1675480': No such file or directory
du: cannot access '/proc/1675510': No such file or directory
du: cannot access '/proc/1675511': No such file or directory
du: cannot access '/proc/1675513': No such file or directory
du: cannot access '/proc/1675514': No such file or directory
du: cannot access '/proc/1675515': No such file or directory
du: cannot access '/proc/1675516': No such file or directory
du: cannot access '/proc/1675522': No such file or directory
du: cannot access '/proc/1675523': No such file or directory
du: cannot access '/proc/1675637': No such file or directory
du: cannot access '/proc/1675640': No such file or directory
du: cannot access '/proc/1675641': No such file or directory
116G    /

Unmounted all shares, no data in these directories:

docker@docker:~$ ls -lah /
total 8.1G
drwxr-xr-x  34 root   root   4.0K Jan  1 12:28 .
drwxr-xr-x  34 root   root   4.0K Jan  1 12:28 ..
drwxr-xr-x   6 docker docker 4.0K Dec 30 11:14 4k_movies
drwxr-xr-x   2 docker docker 4.0K Dec 29 14:24 4k_series
drwxr-xr-x   2 root   root   4.0K Dec 27 13:46 audiobooks
lrwxrwxrwx   1 root   root      7 Jul 31 16:28 bin -> usr/bin
drwxr-xr-x   2 root   root   4.0K Dec 27 12:36 boot
drwxr-xr-x   2 root   root   4.0K Dec 27 12:36 cdrom
drwxr-xr-x   2 root   root   4.0K Dec 27 13:48 cloud
drwxr-xr-x   2 docker docker 4.0K Dec 27 13:47 coding
drwxr-xr-x  19 root   root   4.2K Dec 31 14:01 dev
drwxr-xr-x   2 root   root   4.0K Dec 27 13:48 docker_backup
drwxr-xr-x   2 root   root   4.0K Dec 27 20:10 downloads
drwxr-xr-x   2 docker docker 4.0K Dec 27 13:53 ebooks
drwxr-xr-x   2 root   root   4.0K Dec 27 20:11 elastic
drwxr-xr-x 100 root   root   4.0K Jan  1 13:32 etc
drwxr-xr-x   3 root   root   4.0K Dec 27 13:28 home
drwxr-xr-x   2 docker docker 4.0K Dec 27 13:42 fred
lrwxrwxrwx   1 root   root      7 Jul 31 16:28 lib -> usr/lib
lrwxrwxrwx   1 root   root      9 Jul 31 16:28 lib32 -> usr/lib32
lrwxrwxrwx   1 root   root      9 Jul 31 16:28 lib64 -> usr/lib64
lrwxrwxrwx   1 root   root     10 Jul 31 16:28 libx32 -> usr/libx32
drwx------   2 root   root    16K Dec 27 12:36 lost+found
drwxr-xr-x   2 root   root   4.0K Jul 31 16:28 media
drwxr-xr-x   2 root   root   4.0K Jul 31 16:28 mnt
drwxr-xr-x  17 docker docker 4.0K Dec 30 10:01 movies
drwxr-xr-x   2 docker docker 4.0K Dec 27 13:42 music
drwxr-xr-x   3 root   root   4.0K Dec 27 13:33 opt
drwxr-xr-x   2 docker docker 4.0K Dec 27 13:48 photos
dr-xr-xr-x 457 root   root      0 Dec 31 14:01 proc
drwx------   5 root   root   4.0K Dec 29 22:35 root
drwxr-xr-x  31 root   root    980 Jan  2 14:04 run
lrwxrwxrwx   1 root   root      8 Jul 31 16:28 sbin -> usr/sbin
drwxr-xr-x   6 docker docker 4.0K Dec 30 03:52 series
drwxr-xr-x   6 root   root   4.0K Dec 27 13:28 snap
drwxr-xr-x   2 root   root   4.0K Jul 31 16:28 srv
-rw-------   1 root   root   8.0G Dec 27 12:37 swap.img
dr-xr-xr-x  13 root   root      0 Dec 31 14:01 sys
drwxrwxrwt  13 root   root   4.0K Jan  2 14:04 tmp
drwxr-xr-x  14 root   root   4.0K Jul 31 16:29 usr
drwxr-xr-x  13 root   root   4.0K Jul 31 16:30 var

docker@docker:~$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                                63G     0   63G   0% /dev
tmpfs                               13G  1.5M   13G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  440G  342G   77G  82% /
tmpfs                               63G     0   63G   0% /sys/fs/cgroup
/dev/loop3                          32M   32M     0 100% /snap/snapd/10492
tmpfs                               13G     0   13G   0% /run/user/1000

du -hca --time output:

docker@docker:~$ sudo du -hca --time / --exclude /movies --exclude /series --exclude /photos --exclude /4k_series --exclude /4k_movies --exclude /joshua --exclude /music --exclude /docker_backup --exclude /coding --exclude /cloud --exclude /audiobooks --exclude /ebooks --exclude /downloads --exclude /elastic | grep -E "^[0-9\.]*[G]"
1.2G    2020-12-30 08:59        /usr/lib
2.1G    2020-12-31 13:33        /usr
3.4G    2021-01-04 08:53        /var/log/journal/a5e2984a17e543058168dfd78be80d72
3.4G    2021-01-04 08:53        /var/log/journal
3.5G    2021-01-04 08:53        /var/log
1.4G    2021-01-02 19:54        /var/lib/docker/overlay2/0564cb1f4d542b741588ea217e7d60564172be9b26c56a3721f416bd7a386962/merged/usr
1.5G    2021-01-02 19:58        /var/lib/docker/overlay2/0564cb1f4d542b741588ea217e7d60564172be9b26c56a3721f416bd7a386962/merged/opt/conda/lib
2.1G    2021-01-02 19:59        /var/lib/docker/overlay2/0564cb1f4d542b741588ea217e7d60564172be9b26c56a3721f416bd7a386962/merged/opt/conda
2.6G    2021-01-02 19:59        /var/lib/docker/overlay2/0564cb1f4d542b741588ea217e7d60564172be9b26c56a3721f416bd7a386962/merged/opt
4.0G    2021-01-03 04:02        /var/lib/docker/overlay2/0564cb1f4d542b741588ea217e7d60564172be9b26c56a3721f416bd7a386962/merged
4.0G    2021-01-03 04:02        /var/lib/docker/overlay2/0564cb1f4d542b741588ea217e7d60564172be9b26c56a3721f416bd7a386962
31G     2021-01-04 08:53        /var/lib/docker/overlay2
31G     2021-01-04 08:54        /var/lib/docker
32G     2021-01-04 08:54        /var/lib
35G     2021-01-04 08:54        /var
du: cannot access '/proc/13601/task/13907/net/dev_snmp6/vethbaf5745': No such file or directory
du: cannot access '/proc/1361014': No such file or directory
du: cannot access '/proc/1395411': No such file or directory
du: cannot access '/proc/1409809/task/1409809/fd/4': No such file or directory
du: cannot access '/proc/1409809/task/1409809/fdinfo/4': No such file or directory
du: cannot access '/proc/1409809/fd/3': No such file or directory
du: cannot access '/proc/1409809/fdinfo/3': No such file or directory
8.1G    2020-12-27 12:37        /swap.img
1.9G    2021-01-04 08:54        /home/docker/services/traefik/log/access.log
1.9G    2021-01-04 08:54        /home/docker/services/traefik/log
1.9G    2021-01-04 08:54        /home/docker/services/traefik
5.6G    2021-01-04 08:18        /home/docker/services/jellyfin/data/metadata/library
3.7G    2021-01-04 08:18        /home/docker/services/jellyfin/data/metadata/People
9.4G    2021-01-04 08:18        /home/docker/services/jellyfin/data/metadata
9.8G    2021-01-04 08:20        /home/docker/services/jellyfin/data
11G     2021-01-04 08:20        /home/docker/services/jellyfin
1.1G    2020-12-27 09:31        /home/docker/services/nextcloud/data
1.4G    2020-12-27 09:31        /home/docker/services/nextcloud
1.6G    2021-01-04 08:10        /home/docker/services/tautulli/backups
2.0G    2021-01-04 08:53        /home/docker/services/tautulli
8.8G    2021-01-04 08:18        /home/docker/services/plex/Library/Application Support/Plex Media Server/Metadata/TV Shows
9.3G    2021-01-03 01:14        /home/docker/services/plex/Library/Application Support/Plex Media Server/Metadata/Movies
2.1G    2020-12-27 20:27        /home/docker/services/plex/Library/Application Support/Plex Media Server/Metadata/Albums
21G     2021-01-04 08:18        /home/docker/services/plex/Library/Application Support/Plex Media Server/Metadata
1.3G    2020-12-27 18:50        /home/docker/services/plex/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.blobs.db-2020-12-22
1.3G    2021-01-03 01:01        /home/docker/services/plex/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.blobs.db-2021-01-03
1.3G    2021-01-03 04:03        /home/docker/services/plex/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.blobs.db
1.3G    2020-12-27 18:50        /home/docker/services/plex/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.blobs.db-2020-12-25
1.3G    2020-12-28 01:01        /home/docker/services/plex/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.blobs.db-2020-12-28
8.2G    2021-01-04 08:55        /home/docker/services/plex/Library/Application Support/Plex Media Server/Plug-in Support/Databases
8.5G    2021-01-04 08:55        /home/docker/services/plex/Library/Application Support/Plex Media Server/Plug-in Support
15G     2021-01-04 08:18        /home/docker/services/plex/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder
15G     2021-01-04 08:18        /home/docker/services/plex/Library/Application Support/Plex Media Server/Cache
11G     2021-01-04 08:18        /home/docker/services/plex/Library/Application Support/Plex Media Server/Media/localhost
11G     2021-01-04 08:18        /home/docker/services/plex/Library/Application Support/Plex Media Server/Media
55G     2021-01-04 08:55        /home/docker/services/plex/Library/Application Support/Plex Media Server
55G     2021-01-04 08:55        /home/docker/services/plex/Library/Application Support
55G     2021-01-04 08:55        /home/docker/services/plex/Library
55G     2021-01-04 08:55        /home/docker/services/plex
3.2G    2021-01-04 06:32        /home/docker/services/influxdb/data/telegraf/autogen
3.3G    2021-01-04 07:18        /home/docker/services/influxdb/data/telegraf
3.9G    2021-01-04 08:51        /home/docker/services/influxdb/data
3.9G    2021-01-04 08:55        /home/docker/services/influxdb
4.0G    2021-01-03 15:12        /home/docker/services/radarr/MediaCover
4.4G    2021-01-04 08:55        /home/docker/services/radarr
81G     2021-01-04 08:55        /home/docker/services
81G     2021-01-04 08:55        /home/docker
81G     2021-01-04 08:55        /home
127G    2021-01-04 08:55        /
127G    2021-01-04 08:55        total
14
  • sudo du -hs / then go in to each directory and try again until you find the culprit: cd large_directory; sudo du -hs ./
    – moo
    Commented Jan 1, 2021 at 12:37
  • What about running ncdu with root permissions (sudo)?
    – Murphy
    Commented Jan 1, 2021 at 12:47
  • I have run ncdu as root. I'v ealso added du -hs / in my inital post. For me it seems like some files have not been written on the network shares or mounted disks.
    – In0cenT
    Commented Jan 1, 2021 at 13:04
  • 1
    Unmount all your network drives and disks, and check each directory. This is the most likely cause, that some data was written to a "mount" direcetory without an external drive being mounted. Commented Jan 1, 2021 at 13:05
  • 1
    Experiment: bind-mount / somewhere else (e.g., on /mnt: sudo mount -o bind / /mnt), and check du output on that (sudo du -hd1 /mnt). It's possible that some huge directory is mounted over by something else and not accessible to du directly.
    – muru
    Commented Jan 4, 2021 at 9:27

0

You must log in to answer this question.

Browse other questions tagged .