0

I have Ubuntu machine, version 14.04 running on AWS (EC2). Yesterday, my disk space run out of space. Running with user root

df -h

Bring this result:

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      148G  142G  106M 100% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            3.7G  8.0K  3.7G   1% /dev
tmpfs           748M  332K  748M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.7G     0  3.7G   0% /run/shm
none            100M     0  100M   0% /run/user
overflow        1.0M   32K  992K   4% /tmp

But running this command:

du -ch /

Gave this result:

....
1.5G    /
1.5G    total

Reboot gave save results. This morning, when I came back to the office AND NO ONE TOUCH THE MACHINE (and there is no batch tasks or processes that write and / or delete files), I run again the commands. I got:

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      148G  1.6G  140G   2% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            3.7G  8.0K  3.7G   1% /dev
tmpfs           748M  332K  748M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.7G     0  3.7G   0% /run/shm
none            100M     0  100M   0% /run/user
overflow        1.0M   32K  992K   4% /tmp

Which make much more sense.

Anyone has explanation why this happened? and why after a night space was free?

Please note that I saw the used disk space grow steadily. And it is the third time it happens.

  • I had a similar issue at one time. That was a hidden log file of 94 GB's in size that caused my problem. Specifically .xsessions in my /home. No idea why it might be gone the next day though. – Dan Johansen Oct 20 '14 at 6:04
  • @DanJohansen I don't have such file... where can I find it? – Yonatan Oct 20 '14 at 7:18
  • mine was located in my user directory. In my case /home/strit/.xsessions. I found out it was caused by someone trying to hack into my remote desktop service. – Dan Johansen Oct 20 '14 at 7:20
0

This is just a glitch/bug.

One possible cause is that a temp file/folder/dir was taking up lots of space, but was not cleared yet. When you checked it in the morning, some process cleared the tempspace.

It's nothing to worry about, and probably was just a glitch in the system.


Based on your comment that it was a "hack attempt", that is probably the best answer. During the hack, some glitch or bug was introduced. It also may have just been a large amount of temp files (think memory overflow). Nothing to worry about in the long run.

0

Are you the only person who can login this box?

Why I ask this question is, it looks you are running an application which generate huge log file or other files (for example, turn on the debug option in applications, it fills the space very quickly). Then one guy in your team who delete that file directly, but the space does not released, until you restart the application or reboot the box.

Does that make sense for you?

0

I guess you have already deleted files but they're still open by some program.

Like nginx access.log file. You can delete it, but if the nginx isn't rotated, or restarted. The file will remain.

Try to find all deleted files which is still open.

sudo lsof | grep '(deleted)'

If you find files that are marked as '(deleted)', just restart the program using that files.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.