11

This question arises from another one that I had posed on Stackoverflow. I am using Watcher - the same issues apply to Incron - to monitor a folder and its child folders for changes and silently squirrel away those changes to Dropbox.

I monitor the write_close event - IN_CLOSE_WRITE - for the purpose. Originally I was watching the modify event, i.e. IN_MODIFY. While this worked I found that when writing large files it would fire more than once. That sounded fair so I switched to IN_CLOSE_WRITE since I felt that it was reasonably fair to assume that for a given file it would only occur once.

However, that is not the case. Even for a very small text file - just one character - created in Nano the event occurs two times. At best this can result in unnecessary traffic when the same file is synchronized on Dropbox two times. In my own case it leads to disaster since on the first event I perform the synchronization and then delete the server side file. The result - on the second event the Dropbox side file becomes a 0 byte file.

I am dealing with this for now by making my synchronization script sleep for 10s before I do anything else and then I check that the file in question still exists prior to attempting Dropbox sync. This works because on the second iteration the file is missing and the script just terminates.

This sounds hackish at best. Perhaps not a bad hack but I would prefer to understand - just why does even the IN_CLOSE_WRITE event occur more than one time?


Some additional information

  • Check to ensure that there aren't multiple instances of watcher running.

Output from ps ax|grep watcher.py

23880 ?        Sl     0:01 python /usr/local/bin/watcher.py restart
24977 pts/0    S+     0:00 grep --color=auto watcher.py

The file system is ext4. I should mention that I have encountered precisely the same issue with Incron. I start the Watcher daemon up from a batch script executed via /etc/rc2.d. Incron OTH starts up without any messing about by me via its default apt-get install incron installation.


The essence of my watcher.ini file is shown below.

[DEFAULT]
logfile=/var/log/watcher.log
pidfile=/var/run/watcher.pid

[job1]
watch=/path/to/watch

events=write_close
excluded=
recursive=true
autoadd=true

command=/home/datastore.php $filename

I reduced the datastore.php script to the bare essentials to verify that it is fired up twice without any of my messy Dropbox upload + source delete code.

#! /usr/bin/php
<?php
file_put_contents('/tmp/watcher',$argv[1],FILE_APPEND);

?>

I then created a little file at the path in question and then examined /tmp/watcher. The problem still persists - the file still has two successive entries for $argv[1].

bumped to the homepage by Community 2 days ago

This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.

  • 1
    I've tried many variations, but am unable to duplicate your issue with multiple IN_CLOSE_WRITE firings. Anything I've done causes a single inotify output. I'm going to keep trying things. But so far, only questions. Which file system? Ext4? Other? – lornix Dec 20 '15 at 9:19
  • @lornix - please see the edits to my question. The file system is ext4 and I am reasonably sure I do not have two instances of Watcher running. I have run into the same issue with Incron. – DroidOS Dec 20 '15 at 16:03
  • You said 'I perform the synchronization and then delete the server side file'. Does this delete trigger the second event? Can you disable delete routine and try again? – Germar Dec 20 '15 at 16:16
  • @Germar - see the edit to my question. Even with the sync script doing no real sync'ing and no unlinking the problem persists – DroidOS Dec 21 '15 at 4:28
  • Sorry, rather out of ideas, I cannot reproduce the problem on any of my machines. I get one event, no more. Something else is involved, something not mentioned. Do you have antivirus installed? anything like that? – lornix Dec 21 '15 at 7:07
0

I don't have enough rep to post this as a comment, but are you sure that temporary, possibly-hidden files are not being created? I had a similar issue with inotifywait firing multiple times, but I realized it was because vim would create a .swp file when editing, which would fire an event when closing. It would also pick up the close event from the original file.

It sounds like you're noticing the event firing multiple files on the same file though, which is not something I've been able to reproduce -- this would only happen once for the temp file and once for the original.

I tried a quick test with nano and I don't think it creates a temp file at all (at least for the few character case), but is there anything else in your setup that could rely on similar behavior?

  • thank you for your suggestions. I have run into the inotify multiple issue even when I create a very trivial 1 byte file with Nano - or even by just redirecting a single char from the console into a file. The "solution" I have outlined in my original question is keeping me going for now. However, in the longer term the only solution I have to rebuild my server starting from scratch to identify just when the error begins - my setup with Incron, Watcher (b.t.w it happened when I had just Incron), MariaDB, Nginx, Redis, Memcached... is not exactly a "simple" one. – DroidOS Jan 29 '16 at 10:28
  • Just in case, double check if you are not monitoring the same folder twice. If not, then for example when I copy file to the samba share via os x samba client this happens create,close_write,delete,create,close_write When I do that with a windows client, it looks more reasonable create,write_close and nothing more. So I solve my problem by monitoring the first modification of the file with this IN_MODIFY,IN_ONESHOT /directory. sleep someTime commands oneshot does the thing. – Edik Mkoyan Oct 13 '16 at 13:43
0

I am not sure, but most probably the first write_close writes file attributes into it, like creation time, and only after that it writes actual data. In fact rsync creates a temp file and when everything is done it moves the temp file to the actual file in the same folder so it is easy to monitor has been normally created in when you use rsync, and move is an atomic operation. On the other hand there is something calle one shot in inotify, probably that using that we can trigger something on the first modify message, and as you suggested sleep for reasonable amount of time before starting the operation. I am digging this now, and will update when I find anything new. https://superuser.com/questions/1133642/traceing-the-moment-when-file-is-completely-copied-to-samba-share-with-inotify

  • You may well have put your finger on something quite valid here. It will require some investigation. Thanks for the tip. I will post back in the event that I find that this is somehow an issue. – DroidOS Oct 13 '16 at 9:28
  • I do not think that ATTRIB adds anything to the file itself, I was wrong. – Edik Mkoyan Oct 13 '16 at 11:27

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.