How to find the pid of the process which has deleted a file?

Question

I am working on a project related to VM migration. Sometimes the VM image will disappear and I just want to know who the culprit is. I tried strace on suspicious processes but to no avail.

Answer 1

Finally I found the answer here.

The Linux Audit daemon will do the trick.

sudo auditctl -w /path/to/somefile -p wra

and then

ausearch -f /path/to/somefile -i

Answer 2

You can find out the PID of a process, which has some file open using lsof.

Once file is closed and deleted, you cannot get that information.

BTW. Keep in mind, that deleting a file is operation on directory it's in, not on a file itself.

Answer 3

Let me suggest an alternative with sysdig as the answers above are aging. Let display the pid and name of the processes that delete the file /tmp/test. First we create the file with touch /tmp/test. Then we start sysdig with the following filter:

$ sudo sysdig -p'%proc.pid,%proc.name' '(evt.type=unlinkat and (evt.arg.name=test or evt.arg.name=/tmp/test)) or (evt.type=unlink and evt.arg.path=/tmp/test)'

unlinkat(2) requires an or filter if the path (e.g. evt.arg.name) may be relative. To handle both unlink (which calls unlink(2)) and rm (which calls unlinkat(2) in its GNU version), the filter should match both syscalls.

sysdig should be running when a process deletes the file. Then when we execute such commands:

$ unlink /tmp/test
$ touch /tmp/test
$ rm /tmp/test
$ cd /tmp; touch test; rm test

It will displays such an output:

11380,unlink
11407,rm
11662,rm

Please refer to the sysdig user guide for an explanation about filtering and output.

As the filter is pretty long, I found it convenient to write a chisel. It is a lua script that is associated with a sysdig command:

description = "displays processes that delete a file"
short_description = "spy file deletion"
category = "files"

args = 
{
    {
        name = "path", 
        description = "the path of the file to monitor", 
        argtype = "string"
    },
}

function on_set_arg(name, val)
    path = val
    return true
end

function on_init()
    local filename = path
    for i in string.gmatch(path, "[^/]+") do
        filename = i
    end
    chisel.set_event_formatter("%proc.pid\t%proc.name")
    chisel.set_filter(
        "(evt.type=unlinkat and (evt.arg.name=" .. path .. " or \
                             evt.arg.name=" .. filename .. ")) or \
     (evt.type=unlink and evt.arg.path=" .. path .. ")")
    return true
end

Feel free to comment and improve it. You can put the lua script in a spy_deletes.lua file inside a directory and execute sysdig in this directory to make the chisel available. When typing sudo sysdig -cl you will see it as:

Category: files
---------------
spy_deletes         spy file deletion

Now you can call it:

$ sudo sysdig -c spy_deletes /tmp/test

And in another terminal type:

$ touch test; unlink test
$ touch test; unlink /tmp/test
$ touch test; rm test
$ touch test; rm /tmp/test

It will output:

16025   unlink
16033   unlink
16041   rm
16049   rm

The unlinkat filter would deserve to be more accurate and only match the absolute path. This would require to retrieve the fd of the directory passed to unlinkat(2).