0

Often an automated process will completely fill up the main drive and I won't even be able to login over SSH to fix it. How do I prevent all the space from being used, so that I can always login to my headless server with SSH and delete something?

can't create socket: No space left on device

(There are an infinite number of things that could fill space. A good answer will not care which of them is doing it, but will prevent them from using the space that ssh needs to function.)

  • Allocate more space; cron a job to autoremove; autoclean or remove whatever is being fills your disk space. You haven't provided any detail as to what is filling the space, so we cannot offer you much. – guiverc Apr 2 at 4:58
  • It's something that has never happened to me. I have ~10 Linux computers at home, and ~20 or so at work. Sure, I've had full disks, so full that dhcpd stopped working, but I've been able to login via ssh to clean up... – vidarlo Apr 2 at 5:07
  • It's often the log and cache files that fill up disks. This can be prevented by giving /var an own file system, so that logs can't fill up /. – muclux Apr 2 at 5:30
  • @guiverc It doesn't matter what's filling up the space. What's important is that it shouldn't be possible to take up so much that ssh no longer functions. – endolith Apr 2 at 12:41
  • Interestingly, I just realized that SFTP still works, even though SSH console doesn't, so I was able to delete some files over SFTP and then SSH started working again. – endolith Apr 2 at 14:50
1

Several Linux filesystems, including the most common one ext4, have a feature to deal with this exact problem: by default 5% of the space on the drive is reserved for the root user. This ensures the administrator can always log in to fix out-of-space problems.

TL/DR:

  1. Enable the root account in Ubuntu

  2. Configure the SSH daemon to allow root login

  3. Login with the root account whenever lack of space is preventing normal login

A security note is in order: allowing root logins should be done with care. In particular, it's recommended to allow root login via certificates only, no password logins allowed. In any case, make sure when you define the root password to make it a strong one (it should rarely be needed interactively anyway, so go to town on the password length).

  • Is there no way to do something similar for my user account? – endolith Apr 2 at 14:38
  • @endolith No, the reserved space is always for the root account. – Pedro Lopes Apr 2 at 15:15
-1

Well first try to figure out what is filling up your space. Try to remove any bloatware or background processes which might be filling up your space. You can use: sudo iotop -a or any df commands. Then if there is necessary application which can't be stopped then make a cron job or a habit of regular cleanup of cache and temprary files. There are many ways to clear up your cache.Try:
du -sh /var/cache/apt/archives
sudo apt-get clean
sudo apt-get autoremove --purge
sync; echo 1 > /proc/sys/vm/drop_caches
sync; echo 2 > /proc/sys/vm/drop_caches
sync; echo 3 > /proc/sys/vm/drop_caches

-1

How to prevent running out of disk space?

Fix the problem. If the automated process creates content you need more hard diskl space. If it is creating (error) messages that are logged extra storage is not the correct solution.

Often an automated process will completely fill up the main drive

If you know what the process is there is no need to do anything but for 2 things:

  • clean up the system
  • fix that automated process.

Likely it is a log file in /var/log/ that gets many many many error messages.

If it is a log file make that log file rotate and have it compressed. Put a max size on that log file in place. And the fix the error it is logging.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.