I am using an EC2 instance to run a node app. I logged into the server after a while only to realise that the server has run out of disk space. After debugging, I realised that logs are taking up space. I deleted the 3.3Gb log file. However, even after the cleanup there is no space. What should I do?
Here are the commands I ran:
ubuntu@app1:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 488M 0 488M 0% /dev
tmpfs 100M 11M 89M 11% /run
/dev/xvda1 7.7G 7.7G 0 100% /
tmpfs 496M 8.0K 496M 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 496M 0 496M 0% /sys/fs/cgroup
tmpfs 100M 0 100M 0% /run/user/1001
tmpfs 100M 0 100M 0% /run/user/1000
ubuntu@app1:~$ sudo du -h --max-depth=1 / | sort -n
0 /proc
0 /sys
4.0K /lib64
4.0K /media
4.0K /mnt
4.0K /srv
8.0K /dev
8.0K /snap
16K /lost+found
24K /root
800K /tmp
6.4M /etc
11M /run
14M /sbin
16M /bin
246M /boot
331M /home
397M /opt
429M /var
538M /lib
2.1G /usr
3.7G /data
7.7G /
I deleted a 3.3G log file in /data and ran du again
ubuntu@app1:~$ sudo du -h --max-depth=1 / | sort -h
0 /proc
0 /sys
4.0K /lib64
4.0K /media
4.0K /mnt
4.0K /srv
8.0K /dev
8.0K /snap
16K /lost+found
24K /root
800K /tmp
6.4M /etc
11M /run
14M /sbin
16M /bin
246M /boot
331M /home
352M /data
397M /opt
429M /var
538M /lib
2.1G /usr
4.4G /
ubuntu@app1:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 488M 0 488M 0% /dev
tmpfs 100M 11M 89M 11% /run
/dev/xvda1 7.7G 7.7G 0 100% /
tmpfs 496M 8.0K 496M 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 496M 0 496M 0% /sys/fs/cgroup
tmpfs 100M 0 100M 0% /run/user/1001
tmpfs 100M 0 100M 0% /run/user/1000
Although the /data directory is now reduced to 352M, still df still shows 100% disk utilization. What am I missing?
Referring to this answer https://unix.stackexchange.com/a/253655/47050, here is the output of strace
ubuntu@app1:~$ strace -e statfs df /
statfs("/", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=2016361, f_bfree=4096, f_bavail=0, f_files=1024000, f_ffree=617995, f_fsid={2136106470, -680157247}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda1 8065444 8049060 0 100% /
+++ exited with 0 +++
=
After running sync && sync && sync,
df -h still shows
/dev/xvda1 7.7G 7.7G 0 100% /.
But sudo du -h shows 4.4G
=
Then ran sudo lsof | grep deleted and found many lines like
node\x20/ 22318 deploy 12w REG 202,1 3541729280 791684 /data/app/shared/logs/production.log (deleted)
How do I release these files?
