Unable to free up disk space

Question

I used Kazam to record a 12-minute screen recording and it somehow managed to create a 40 GB file. This caused my computer (really the computer on a Husky robot) to run out of memory. I tried to delete the files, but there wasn't enough memory left to do that.

I read a couple of tutorials and askubuntu posts and saw that it was recommended to boot with a Live USB and delete the files that way.

I was able to do this and used sudo nautilus to give me the ability to remove the large video files.

However, the computer still didn't have enough memory to run the robot's startup scripts. I was able to SSH in and run sudo tune2fs -m 0 /dev/sda6 (as another post suggested) and I was able to make enough memory to actually boot the computer fully. I then proceeded to make sure all the Trash folders for every user where fully emptied and manually did rm -rf ~/.local/share/Trash/*.

Now, df still has /dev/sda6 as being almost full:

eric@cpr-mic09:~$ df
Filesystem     1K-blocks      Used Available Use% Mounted on
udev             4014820         0   4014820   0% /dev
tmpfs             807508      2892    804616   1% /run
/dev/sda6      114756748 109069280   5671084  96% /

I tried using du to see if there were any obviously large files, but nothing looked out of the ordinary. I also used lsof to try to find deleted files sudo lsof | grep deleted (source). I even went so far as to just kill everything that lsof returned (source):

$ kill -9 $(lsof -ti | awk "{print $2}")

Nothing I have done so far has been able to get the Disk Space to free up. I've rebooted multiple times to no avail. Below are some relevant screenshots. I would really appreciate any advice.

Edit:

The inode usage looks okay. I checked with df -i and /dev/sda6 is only using 16%.

Other sources I used:

• No more disk space: How can I find what is taking up the space?
• Large discrepancy between df and du
• https://serverfault.com/questions/62411/how-can-i-sort-du-h-output-by-size
• https://phoenixnap.com/kb/linux-check-disk-space
• df shows all space taken, but du doesn't add up
• https://unix.stackexchange.com/questions/182077/best-way-to-free-disk-space-from-deleted-files-that-are-held-open
• https://unix.stackexchange.com/questions/68523/find-and-remove-large-files-that-are-open-but-have-been-deleted
• https://unix.stackexchange.com/questions/34140/tell-fs-to-free-space-from-deleted-files-now
• How do I free up disk space?

Answer 1

I was able to fix this after many, many hours. It was a combination of quite a few issues.

Issue #1: Deleting files using a Live USB

I had booted using a Live USB to try to delete files. In hindsight, it would have been better to just SSH in and manually rm -rf the files.

Because I booted with a Live USB to delete files, the files weren't fully deleted. Instead, they were moved to a file /.Trash-0. This is because:

Whenever you mount an external disk, a trash-* directory will be created in the "root" directory where files are moved to if deleted from the GUI, like the trash on your desktop.

This is to improve file recovery for external drives. (E.g. USB sticks or HDDs)

Source

So to sum that up, when using a Live USB to delete files, the files will actually be moved into /.Trash-0 instead of actually being deleted.

Issue #2: trash-cli

Someone in the lab had added the alias:

alias rm = 'trash-put`

trash-put is a tool that moves files into a Trash folder from the command line. Unlike rm, which immediately and permanently removes a file, trash-put allows you to restore the files from the Trash folder.

When the tool was first released, it was recommended to make this change. However, the author now strongly recommendss not to make this alias source.

You can avoid using the alias rm if someone has set it by doing \rm instead of rm.

How I figured this out

I was trying to use lsof to find a file that was being held open by a process and not allowed to be deleted. However, this didn't help me because the file wasn't being held open. It was just sitting inside a different trash folder.

I was able to finally find where it was by doing:

# Switch to the admin account
$ sudo su administrator

# Move into the root directory
$ cd /

# List the top ten largest files
$ sudo du . | sort -nr | head -n10

I did this and the largest file was ./Trash-0 followed by ./.Trash-0/files.

I was able to delete the files by doing sudo \rm -rf ./.Trash-0 (for people without an aliased rm, you can just do sudo rm -rf ./.Trash-0).

Takeaways

Some lessons I learned from this experience:

• Make backups
• Don't use Kazam to make screen recordings
• After working to solve a problem for 8 hours, you should probably come back to it fresh
• Using a Live USB to delete files isn't the best move
• Don't build a $300,000 research robot with only 128 GB of memory
• To give yourself enough memory for basic functionality, you can use sudo tune2fs -m 0 /dev/sda6 to allow you to use memory preserved for essential functions. Make sure to restore this to sudo tune2fs -m 5 /dev/sda6 when you're done.