You've got some out-of-control logs. Instead of deleting like crazy everyday, find the fast growing file or files, and look inside to investigate what may be causing this. Maybe some program is spinning in a loop logging some condition. Either disable that program, disable its logging or try to fix the condition that it's complaining about.
If a file is growing before your eyes, and you have no idea which program is writing to it, you may be able to find that out easily. Here is an example. Who has /var/log/syslog
open? We use the fuser
command:
# fuser /var/log/syslog
/var/log/syslog: 602
Only one process has /var/log/syslog
open. It is process 602. What is that? Let us not bother with ps
and grep
, but look at the /proc
filesystem directly:
# ls -l /proc/602/exe
lrwxrwxrwx 1 root root 0 Mar 29 17:45 /proc/602/exe -> /usr/sbin/rsyslogd
Aha, it is rsyslogd
. We are not surprised that rsyslogd
has /var/log/syslog/
open.
This method is not guaranteed to work. The reason is that programs do not have to keep files open ino rder to write to them. Suppose you have a process which opens a file, appends to it, and then closes it. You will have a somewhat more difficult investigation. You could run fuser
many times until by chance you catch the process "red handed". That process itself could be going into and out of existence quickly. Another problem is that multiple processes could have the file open, but only one is making it larger. In that case, you can trace their system calls.
# fuser /var/log/huge-annoying-file
/var/log/huge-annoying-file: 1234 23459
Oops! Two processes have it open: 1234 and 23459. Let's see what they are doing:
# strace -p 1234
Process 1234 attached - interrupt to quit
select(1, NULL, NULL, NULL, {9, 922666}
It's not doing anything, just blocking in a select
call. Ctrl-C to break the trace:
select(1, NULL, NULL, NULL, {9, 922666}^C <unfinished ...>
Check the next one:
# strace -p 23459
write(5, "Useless garbage ..."..., 512) = 512
write(5, "More useless garbage ..."..., 512) = 512
write(5, "More useless garbage ..."..., 512) = 512
write(5, "More useless garbage ..."..., 512) = 512
write(5, "More useless garbage ..."..., 512) = 512
write(5, "More useless garbage ..."..., 512) = 512
write(5, "More useless garbage ..."..., 512) = 512
^C
Oops, that one is writing constantly. It must be the bad one. We can even check that the file descriptor 5 which the process is writing to is in fact the large file:
# ls -l /proc/23459/fd/5
lr-x------ 1 root root 64 Apr 3 23:39 /proc/23459/fd/5 -> /var/log/huge-annoying-file
I don't suspect you have a corrupt filesystem, but to force a full check, you don't have to boot a DVD.
Firstly, review your filesystem's maximum mount count setting. Identify your partition using the df command. Example on an Ubuntu system I have here:
# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 18062108 5499320 11645284 33% /
udev 392152 4 392148 1% /dev
tmpfs 159768 768 159000 1% /run
none 5120 0 5120 0% /run/lock
none 399416 200 399216 1% /run/shm
/dev/sr0 43668 43668 0 100% /media/VBOXADDITIONS_4.1.4_74291
You can see that the /
filesystem is mounted on /dev/sda1
. So /dev/sda1
is the storage device of the root partition (and the only partition in this particular system).
Let's look at some attributes of that filesystem. This is safe to do even though it is mounted. The command spews a lot of output. Here is an excerpt:
$ dumpe2fs /dev/sda1
dumpe2fs 1.42 (29-Nov-2011)
Filesystem volume name: <none>
Last mounted on: /
[ ... SNIP ... ]
Last mount time: Fri Mar 29 17:45:18 2013
Last write time: Tue Mar 5 09:08:03 2013
Mount count: 22
Maximum mount count: 22
[ ... SNIP ... ]
Hey look, the mount count is equal to the maximum mount count. Next time I reboot, there will be a filesystem check. The important thing is that the mount count is a positive value. If yours is zero, change it to some positive value like 22 using tune2fs -c 22 /dev/whatever
. Zero means that a check is never forced regardless of how many times the partition is mounted. Rarely rebooted systems should have low values here. A server that goes down once a year could probably use a fsck each time it reboots. You can set date-based check intervals also.
Now to force a check, you can override the actual count to be greater than or equal to the maximum, and then reboot. That's done with capital C
: tune2fs -C 1234 /dev/whatever
. Now the partition looks like it has been mounted 1234 times without a check, which is greater than the one- or two-digit maximum.
sudo du -sh /var/* ~/.xsession-errors
please? (those two places I would expect to blow up if there is something being silly). Otherwise, I'm with Eliah - this is indicative of disk issues. Take this seriously.