12

Is there a tool to monitor what processes open what files on the system so you can track down which process keeps touching a specific file?

Lsof can find out if you run it while the process has the file open, but if it is a short lived process that runs every once in a while, you can't catch it with lsof. Need something that uses kernel tracing.

3
  • Have you checked out inotify? See @Kees's answer here for example: askubuntu.com/questions/25442/… There are a couple of links on my answer here: askubuntu.com/questions/29566/…
    – belacqua
    Apr 8, 2011 at 20:13
  • @jgbelacua neither of those is quite what I'm looking for. Inotify can tell you when a given file is touched, and lsof can tell you what files a process has open, or what process has a file open, but I need to figure out what process keeps touching a file, then closing it before I can run lsof to catch it.
    – psusi
    Apr 9, 2011 at 3:02
  • Related: askubuntu.com/questions/24512/…
    – ændrük
    May 14, 2011 at 22:44

3 Answers 3

8

You could perhaps use audit system for that. It is a little heavyweight, but something like this should work (in /etc/audit/audit.rules):

# delete all other rules
-D

# watch the file in question
-w /path/to/file -p rwxa

and then I think you need to restart auditd:

sudo service audit restart

(In case you don't have it installed, it is in package auditd.) The culprit can then be found in /var/log/audit/audit.log.

1
  • Perfect! That is exactly what I was looking for.
    – psusi
    Jul 2, 2011 at 0:24
1

fnotifystat is a tool that has been designed to watch linux file activity

sudo apt-get install fnotifystat
sudo fnotifystat
Total   Open  Close   Read  Write   PID  Process         Pathname
  7.0    1.0    1.0    5.0    0.0   2075 libvirtd        /proc/cpuinfo
  6.0    2.0    2.0    2.0    0.0   2075 libvirtd        /sys/devices/system/cpu/cpu0/topology/physical_package_id
  6.0    2.0    2.0    2.0    0.0   2075 libvirtd        /sys/devices/system/cpu/cpu1/topology/physical_package_id
  6.0    2.0    2.0    2.0    0.0   2075 libvirtd        /sys/devices/system/cpu/cpu2/topology/physical_package_id
  6.0    2.0    2.0    2.0    0.0   2075 libvirtd        /sys/devices/system/cpu/cpu3/topology/physical_package_id
  4.0    1.0    1.0    2.0    0.0   2075 libvirtd        /sys/devices/system/node
  4.0    1.0    1.0    2.0    0.0   2075 libvirtd        /sys/devices/system/node/node0
  4.0    2.0    2.0    0.0    0.0  15313 gnome-calendar  /usr/share/zoneinfo/Europe/London
  3.0    1.0    1.0    1.0    0.0   2075 libvirtd        /sys/devices/system/cpu/cpu0/topology/core_id
  3.0    1.0    1.0    1.0    0.0   2075 libvirtd        /sys/devices/system/cpu/cpu0/topology/thread_siblings_list
  3.0    1.0    1.0    1.0    0.0   2075 libvirtd        /sys/devices/system/cpu/cpu1/topology/core_id
  3.0    1.0    1.0    1.0    0.0   2075 libvirtd        /sys/devices/system/cpu/cpu1/topology/thread_siblings_list
  3.0    1.0    1.0    1.0    0.0   2075 libvirtd        /sys/devices/system/cpu/cpu2/topology/core_id
  3.0    1.0    1.0    1.0    0.0   2075 libvirtd        /sys/devices/system/cpu/cpu2/topology/thread_siblings_list
  3.0    1.0    1.0    1.0    0.0   2075 libvirtd        /sys/devices/system/cpu/cpu3/topology/core_id
  3.0    1.0    1.0    1.0    0.0   2075 libvirtd        /sys/devices/system/cpu/cpu3/topology/thread_siblings_list
  3.0    1.0    1.0    1.0    0.0   2075 libvirtd        /sys/devices/system/cpu/online
  3.0    1.0    1.0    1.0    0.0   2075 libvirtd        /sys/devices/system/cpu/present
  3.0    1.0    1.0    1.0    0.0   2075 libvirtd        /sys/devices/system/node/node0/meminfo
  2.0    0.0    0.0    0.0    2.0  12174 xchat           /home/cking/.xchat2/xchatlogs/FreeNode-#ubuntu-release.log
  1.0    0.0    0.0    0.0    1.0  12174 xchat           /home/cking/.xchat2/xchatlogs/FreeNode-#ubuntu-desktop.log
  1.0    0.0    0.0    0.0    1.0  12174 xchat           /home/cking/.xchat2/xchatlogs/FreeNode-#ubuntu-devel.log
  1.0    0.0    0.0    0.0    1.0  12174 xchat           /home/cking/.xchat2/xchatlogs/FreeNode-#ubuntu-kernel.log

Show the top 10 active files every 60 seconds until stopped:

sudo fnotifystat -t 10 60

Show file acivity every 10 seconds just 6 times:

sudo fnotifystat 10 6

Show file activity of thunderbird and process ID 1827:

sudo fnotifystat -p thunderbird,1827

Show every file notify event and the top 20 active activity files over a single period of 5 minutes:

sudo sudo notifystat -v -d -c 5m 1

Just show every file notify event on /sys and /proc and no periodic statisics:

sudo fnotifystat -n -i /sys,/proc

Consult the fnotifystat man page for more information, it's quite a flexible tool.

0

Unfortunately the mechanism Linux uses to allow one to monitor files is inotify, which does not provide enough information to extract useful data: you only get the file name and the action that was done.

I've tried using something like this:

sudo inotifywait -mr somedir --format "%w%f" | while read file; do echo -n "$file => ";lsof -b $file; echo ""; done

This listens to inotify events on the specified directory and for each event it runs lsof to try to catch the process that touches the file. Unfortunately for most accesses that I tested (such as using an editor to write to a file) the LSOF command is just to slow and doesn't manage to catch the offending process.

If your processes do some more intensive IO on the problematic files, then your mileage may vary. Good luck.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .