XFS root partition free space inconsistency

Question

some time ago (I think it started after my upgrade to 16.04) I started receiving notifications about going low with free space on the root partition. First I thought that it might be due to packages kept within /var/cache after the upgrade, so I moved the folders to another partition and sym linked them to the original location. This helped but not for long. Then I noticed that I have quite a few old kernels and associated headers so I cleaned them up. This gave me about 1GB extra space, which then again disappeared. These are my mountpoints and their current usage as shown by df -h:

Filesystem      Size  Used Avail Use% Mounted on
udev            981M     0  981M   0% /dev
tmpfs           200M   19M  181M  10% /run
/dev/sda6       8.0G  8.0G   16M 100% /
/dev/sda7        32G   16G   17G  49% /usr
tmpfs           999M  300K  999M   1% /dev/shm
tmpfs           5.0M  8.0K  5.0M   1% /run/lock
tmpfs           999M     0  999M   0% /sys/fs/cgroup
/dev/sda9       270G   41G  216G  16% /repo
/dev/sda8        99G   91G  2.6G  98% /home
/dev/sda10      504G  162G  317G  34% /mnt/media
cgmfs           100K     0  100K   0% /run/cgmanager/fs
tmpfs           200M   64K  200M   1% /run/user/1000
/dev/sr2         48M   48M     0 100% /mnt/usb-HUAWEI_Mass_Storage-0:0

As you can see / (/dev/sda6) is practically full. When I run sudo du -h -x --max-depth=1 / I get:

469M    /var
24M     /etc
0       /media
13M     /bin
54M     /boot
326M    /lib
198M    /opt
6.9M    /root
15M     /sbin
0       /srv
8.0K    /tmp
0       /cdrom
0       /mnt
0       /snap
1.1G    /

which as far as I can tell shows me far less than 8GB being used. I have been using linux on my desktop since the 90's, but apparently I am missing something here that I cannot explain. How do I find out what is consuming space on my /?

[EDIT]: I tried solution presented here: https://serverfault.com/questions/406069/why-are-my-xfs-filesystems-suddenly-consuming-more-space-and-full-of-sparse-file (allocsize=64k option), but no space was freed.

[EDIT 2]: Per Andrius's comment, output from xfs_db:

magicnum = 0x58465342
blocksize = 4096
dblocks = 2096474
rblocks = 0
rextents = 0
uuid = 3a3ee397-d4e5-40f1-b179-94f3c0b566ac
logstart = 1048580
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 1
agblocks = 524119
agcount = 4
rbmblocks = 0
logblocks = 2560
versionnum = 0xb4b4
sectsize = 512
inodesize = 256
inopblock = 16
fname = "u.root\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 19
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 60800
ifree = 27330
fdblocks = 2970
frextents = 0
uquotino = nic
gquotino = nic
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 1
features2 = 0x8
bad_features2 = 0x8

Answer 1

This is quite interesting but it is that xfs keeps metadata about the files in the xfs filesystem internally but this does not seem to be picked up by du that much but is by df. However xfs has its own xfs_estimate command. df properly shows the free space used on the xfs partition. I have found that running sudo du -h -x --max-depth=0 /path ; sudo xfs_estimate /path sudo can be nesecary in case there are files your normal user cannot see like kvm/qemu virtual machines by defualt which are one thing in / that do take up a lot of space. For example when I run sudo du -h -x --max-depth=0 /path ; sudo xfs_estimate /path on /etc with etc having lots of small configuration files This shows /etc as taking up 13 MB of space but xfs estimate shows it will need a 20 MB partition which would seem like you are using 65 % on your du command but in reality the metadata is using the rest. So in the end xfs_estimate can tell you a lot.

Answer 2

I ran xfs_check and it showed that there is more space allocated than being used. I tried xfs_repair and it fixed the issue. From what I understand, the problem was similar to what was sometimes occuring with FAT partitions which were prone to 'lost clusters' errors. Nonetheless, I still don't know what was responsible for that and how to precent similar issues in future.

Answer 3

I am afraid your XFS is not in "lazy-counter" mode, there are some mistake in it. So please make a try to change your XFS to "lazy-counter" mode through command "xfs_admin -c 1".