I've got an Ubuntu 12.04.2 server whose root partition filled up. I discovered that the /var/log/auth.log file had grown to around 3.8GB. After compressing this file using gzip, the file size was around 110MB.

Running the df -h command, the root partition's disk usage is still at 100%. I'm assuming a reboot will correct this, but since it's a production server, I was wondering if there was a way to convince this server that it actually has much more free space than it thinks.

Thanks in advance for the help and advice.

EDIT: Lesson learned: lsof is your friend!

After running lsof and grepping for auth.log, I was able to determine that another process was using auth.log. After restarting that process, the free space was recognized by the server.

  • Depending on the actual size of the partition these 3.7 GB might not matter much – Gerald Schneider Aug 2 '13 at 13:05
  • Sorry, I probably should have posted this in the question. It's only an 8 GB partition... we use Amazon Elastic Block Store volumes and attach them to our servers as needed. – arochette Aug 2 '13 at 13:14
  • can you post the output of df -i and tune2fs -l /dev/<your root device>? – Gerald Schneider Aug 2 '13 at 13:19
  • Since I'll be off for a while soon a last hint I can think of: Filesize is not the only thing filling up a hard disk. If you produce enough 0 byte files you can fill it too, because every file uses an inode and a block. Additionally, blocks are reserverd for system use. You can check the usage of both with the commands mentioned above. – Gerald Schneider Aug 2 '13 at 13:52
  • 1
    Thanks for the advice. Since I don't have 10 reputation, I am not able to answer my own question until 8 hours after posting. Once that time has elapsed, I will answer it in detail and mark it complete. – arochette Aug 2 '13 at 14:48

To figure this out, I ran sudo lsof | grep auth.log. This gave me a list of services that were using the auth.log file.

Rsyslog was one that I had already figured would have auth.log open, so I had already restarted it. What I forgot to take into account was the fact that we use OSSEC, and it had auth.log open as well. Restarting OSSEC solved the problem; the free space was then recognized by the server.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.