Investigate disk writes further to find out which process writes to my SSD

Question

I try to minimize disk writes to my new SSD system drive. I'm stuck with iostat output:

~ > iostat -d 10 /dev/sdb
Linux 2.6.32-44-generic (Pluto)     13.11.2012  _i686_  (2 CPU)

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sdb               8,60       212,67       119,45   21010156   11800488

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sdb               3,00         0,00        40,00          0        400

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sdb               1,70         0,00        18,40          0        184

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sdb               1,20         0,00        28,80          0        288

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sdb               2,20         0,00        32,80          0        328

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sdb               1,20         0,00        23,20          0        232

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sdb               3,40        19,20        42,40        192        424

As I see there are writes to sdb. How can I resolve which process writes?

I know about iotop, but it doesn't show which filesystem is being accessed.

Answer 1

You could at least start with iotop. It won't tell you which filesystem is being written but it will give you some processes to investigate.

sudo apt-get install iotop
sudo iotop

It shows instantaneous disk reads and writes and the name of the command reading or writing.

If you are trying to catch a process that writes infrequently, you can use the --accumulated option or log the output to a file:

sudo -i
iotop --batch > iotop_log_file

Obviously the writing of the log file will show up in the results, but you should also be able to grep for other processes writing to disk.

By this point you should be able to find some candidate suspect processes. The left column in iotop shows the pid. Next, find out which file descriptor the process is writing to:

sudo -i
strace -p <pid> 2>&1 | grep write

You should see output like this when the process writes:

write(1, "\n", 1)                       = 1
write(4, "test\n", 5)                   = 5
write(1, ">>> ", 4)                     = 4

The first argument to write is the file descriptor. We are probably looking for values greater than 2, because 0, 1 and 2 are just stdin, stdout and stderr. File descriptor 4 looks interesting.

You can now find out what file the file descriptor points to with:

lsof -p <pid>

Which should yield output like:

...
python  23908  rob  mem    REG    8,1    26258 8392656 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
python  23908  rob    0u   CHR  136,5      0t0       8 /dev/pts/5
python  23908  rob    1u   CHR  136,5      0t0       8 /dev/pts/5
python  23908  rob    2u   CHR  136,5      0t0       8 /dev/pts/5
python  23908  rob    3w   REG   0,25      909 9049082 /home/rob/testfile
python  23908  rob    4w   REG   0,25       20 9049087 /home/rob/another_test_file

Look at the 4th column. 4w means that file descriptor 4 is open for writing and the file is another_test_file.

It is possible for a process to open, write and then close a file, in which case lsof would not show it. You might catch this happening with:

strace -p <pid> 2>&1 | grep open

Answer 2

The following uses the kernel's virtual memory block dump mechanism. First get the perl script:

wget https://raw.githubusercontent.com/true/aspersa-mirror/master/iodump

Then turn on block dump:

echo 1 | sudo tee /proc/sys/vm/block_dump

And run the following:

while true; do sleep 1; sudo dmesg -c; done  | perl iodump

..and press Controlc to finish, you will see something like the following:

^C# Caught SIGINT.
TASK                   PID      TOTAL       READ      WRITE      DIRTY DEVICES
jbd2/sda3-8            620         40          0         40          0 sda3
jbd2/sda1-8            323         21          0         21          0 sda1
#1                    4746         11          0         11          0 sda3
flush-8:0             2759          7          0          7          0 sda1, sda3
command-not-fou       9703          4          4          0          0 sda1
mpegaudioparse8       8167          2          2          0          0 sda3
bash                  9704          1          1          0          0 sda1
bash                  9489          1          0          1          0 sda3
mount.ecryptfs_       9698          1          1          0          0 sda1

And turn off block dump when you are finished:

echo 0 | sudo tee /proc/sys/vm/block_dump

Thanks to http://www.xaprb.com/blog/2009/08/23/how-to-find-per-process-io-statistics-on-linux/ for this helpful info.