35

Ubuntu server shows me that I use almost all disk:

Usage of /:   95.5% of 118.12GB

And I try to find big folders and files, run ncdu:

ncdu 1.8 ~ Use the arrow keys to navigate, press ? for help                                                                                                                                                 
--- / ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    5.5GiB [##########] /root                                                                                                                                                                               
    2.3GiB [####      ] /var
  628.6MiB [#         ] /usr
  209.9MiB [          ] /lib
   28.2MiB [          ] /boot
    8.6MiB [          ] /bin
    7.7MiB [          ] /sbin
    6.6MiB [          ] /etc
  208.0KiB [          ] /run
  112.0KiB [          ] /tmp
   48.0KiB [          ] /opt
e  16.0KiB [          ] /lost+found
    8.0KiB [          ] /dev
    8.0KiB [          ] /media
    4.0KiB [          ] /lib64
e   4.0KiB [          ] /srv
e   4.0KiB [          ] /selinux
e   4.0KiB [          ] /mnt
e   4.0KiB [          ] /home
    0.0  B [          ] /proc
    0.0  B [          ] /sys
@   0.0  B [          ]  initrd.img
@   0.0  B [          ]  vmlinuz

According to ncdu I use about 10 GiB of 128 GiB - it is about 10 %. Contradiction.

How to clean my ubutntu server without rebooting?

I thought that ncdu lies and used another apps to find big files and folders. All of them shows the same result as ncdu.

And df -h command shows that disk is full.

# df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda       119G  113G     0 100% /
udev            2.0G  8.0K  2.0G   1% /dev
tmpfs           788M  212K  788M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.0G     0  2.0G   0% /run/shm

Update

sudo du -sch /* result:

$ sudo du -sch

8.7M    /bin
29M /boot
8.0K    /dev
6.6M    /etc
4.0K    /home
0   /initrd.img
210M    /lib
4.0K    /lib64
16K /lost+found
8.0K    /media
4.0K    /mnt
48K /opt
du: cannot access `/proc/4470/task/4470/fd/4': No such file or directory
du: cannot access `/proc/4470/task/4470/fdinfo/4': No such file or directory
du: cannot access `/proc/4470/fd/4': No such file or directory
du: cannot access `/proc/4470/fdinfo/4': No such file or directory
0   /proc
5.0G    /root
212K    /run
7.8M    /sbin
4.0K    /selinux
4.0K    /srv
0   /sys
112K    /tmp
629M    /usr
2.3G    /var
0   /vmlinuz
8.1G    total

8.1G total as usual. But I see cannot access rows, maybe problem because of them.

Then I checked the biggest folder in /. It is /root :

$ sudo du -sch /root

96K /root/Downloads
2.5G    /root/Dropbox
36K /root/nohup.out
4.0K    /root/npm-debug.log
4.0K    /root/readonly
980K    /root/redis-2.6.16.tar.gz
228M    /root/tmp
2.7G    total
8
  • Just a thought, might check the contents of /var/log/ to see if any logs have grown expotenially.
    – Mordoc
    Jan 31, 2014 at 4:50
  • /var/log is about 2 GiB. It is ok Jan 31, 2014 at 4:53
  • 1
    Try du -sch /* to see which root directories are using the most space, and descend from there into the places using the most space.
    – DopeGhoti
    Jan 31, 2014 at 5:21
  • @DopeGhoti I tried but saw the same about 8.1 GiB full (added this to update). Cannot figure out where is the rest about 100 GiB Jan 31, 2014 at 5:36
  • 2
    I know you don't want to, but bite the bullet and reboot.
    – douggro
    Jan 31, 2014 at 8:23

10 Answers 10

21

I was running into this same issue on our lab machines and using this command

du -sch .[!.]* * |sort -h

I was able to find hidden files like inside of users trash bins that they had yet to delete.

Credit to here where I originally found this answer.

1
  • Amazing solution!
    – AivanF.
    Nov 4, 2018 at 15:08
17

Check for deleted files that are still being held open by a process:

sudo lsof | grep deleted | less

That'll show the pid and file descriptor. I had this exact problem on a server, nothing in ncdu but disk filling up. It turned out to be a nightly process that moved files to a mounted samba share and occasionally didn't close out the file handle correctly, it seems.

If you find deleted files and want to clean them up, a reboot is probably easiest if that's acceptable. Or you can try killing the process. Or if you're sure they're not being used, you can manually empty them out, with something like this:
> /proc/14487/fd/12

3
  • This was my issue. Tomcat was holding an 80GB deleted file. A restart was enough to fix it.
    – AFP_555
    Sep 23, 2019 at 13:41
  • 1
    How can I delete them if the "reboot" command it's not enough?
    – sparkle
    Oct 15, 2019 at 11:52
  • 1
    This shows many files what would I do with this info? I already rebooted and it didn't do anything.
    – Kvothe
    Sep 15, 2022 at 15:13
5

We had this same issue and it turned out to be docker images, stored under var/lib/docker

ncdu does not list these as they are not visible to users. even running ncdu under sudo does not help.

This command purges all existing docker images...

docker rmi $(docker images -a -q)

2
  • 1
    Same issue here. In fact, even docker system prune wasn't finding everything. This command (which predates docker system prune) does the trick.
    – jscharf
    Dec 17, 2018 at 22:13
  • 3
    recently we have discovered that docker system prune -a -f is a lot more thorough
    – Baldy
    Dec 20, 2018 at 11:52
4

Following command will show disk utilization for /home directory with --max-depth=1:

sudo du -h -d 1 /

From manual:

-d, --max-depth=N print the total for a directory (or file, with --all) only if it is N or fewer levels below the command line argument; --max-depth=0 is the same as --summarize.

3

Be sure to check your disk mounts. None of the solutions I've seen here can identify space being taken up by a folder which has a mount placed over it.

3
  • any suggestion?i think this might be my problem Jan 23, 2018 at 5:02
  • 1
  • Basically, check your existing mounts with mount, then add a second mount for each of the directories which have mounts placed over them. Then you can use normal disk tools like du on the newly created mount to see if it's the culprit.
    – Rich Remer
    Jan 25, 2018 at 16:01
3

It's possible that a process has opened a large file which has since been deleted. You'll have to kill that process to free up the space. You may be able to identify the process by using lsof. On Linux deleted yet open files are known to lsof and marked as (deleted) in lsof's output.

You can check this with sudo lsof +L1

please refer: https://serverfault.com/questions/315181/df-says-disk-is-full-but-it-is-not

1

One reason for "hidden" big files is a possibility to mount a folder (to some other folder/resource residing on a different disk) - the whole content of the original folder is kind of "hidden" - "du" won't show it, because it will be "overlapped" with the mounted resource.

0
1

I was experiencing this issue myself. Disk full, completely unable to locate the offending directories and/or files. However, I did solve this puzzle.

All of the previous answers are extremely helpful and legitimate but if you are, like me, unable to find the offending directories and/or files using the tips above, I'd like to share my solution.

I have an external USB drive (ex: dev/sdeX) that I used exclusively for backing up the content of root (ex: /dev/sdaX). Using that as a clue, I unmounted all mount points, excluding root and other required mount points. From there, I noticed the mount point to which the external USB drive mounts to is utilizing over 150G of space. Sure enough, I venture to this directory and see that it contains the same exact file structure and content as the external USB drive (when mounted). I mounted my external USB drive to verify backup data is intact (it is, check) then unmounted it and deleted the contents of that directory (for peace of mind, make sure to verify the mount point has been unmounted). Once I did that, usage on / (/dev/sdaX) went from 100% usage to 13%, which is right about what I expect.

Point of my answer... try unmounting all external drives and double check the mount point for data that should be on the external drive, not your root drive (/dev/sdaX).

Hope this helps.

0

You can run the command below to find the top 10 files by disk usage:

find / -type f -printf '%s %p\n' 2>&1 \
     | grep -v 'Permission denied' \
     | sort -nr \
     | head -10
1
  • find / -type f -printf '%s %p\n' 2>&1 \ | grep -v 'Permission denied' \ | sort -nr \ | head -10 this command is not working
    – Shabby
    Jul 22, 2022 at 7:20
0

switch to root user than run ncud on required folder, it will work. due to permission issue it was not showing files taking space. It's in my case, maybe something else works for you.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .