I have an annoyingly complicated find command I've cobbled together that I pipe to rsync with the intention of copying files and syncing new versions. I'm trying to write something that is efficient enough to be run in cron.

find /home/archive/dir1/ -not -path '*/\.*' -ipath "/home/archive/dir1/*/*document-name*" -type f \( -iname "*.ods" -o -iname "*.pdf" \) -print0 | rsync -v --delete --files-from=- --from0 / /home/user/newdir2/

The idea is that it goes into a huge directory, pulls out the files that have filenames with document-name in the path.

The problem is it doesn't just copy files, it recursively copies all directory structure, when I simply want a all files returned from find to be collected into this separate directory.

I get exactly the result I want with:

find /home/archive/dir1/ -not -path '*/\.*' -ipath "/home/archive/dir1/*/*document-name*" -type f \( -iname "*.ods" -o -iname "*.pdf" \) -exec rsync -avz --delete {} /home/user/newdir2/ \;

Trouble is this spawns a billion rsync instances to do something that a single file list ought to do, probably not great for cron.

IE, this:

/home/archive/dir1/New1/New1 document-name.ods
/home/archive/dir1/New2/New2 document-name.pdf


/home/user/newdir2/New1 document-name.ods
/home/user/newdir2/New2 document-name.pdf

Any suggestions on improving my command welcome.

I have tried using exclude in rsync, but I do not think it's the right solution.

How can I efficiently sync all of these files?

  • Your find definitly not matches the dir[1-9234] given in your example. Are all files you want to search in dir1? As a side note: -ipath is depricated and should be avoided. – Marcus Feb 12 '16 at 0:08
  • Thank you for pointing that out. My example was actually written out wrong, the command did exactly what I wanted it to when I ran it with -exec, so I knew that was at least CORRECT. – ezgoodnight Feb 12 '16 at 2:59
  • Also @Marcus, what do you recommend instead of -ipath or -path? – ezgoodnight Feb 12 '16 at 3:07
  • there is another bit: /archive/dir1/... in ipath must have the same root as the find location /home/archive/dir1/ to work properly. I assume it is -ipath "/home/archive/dir1/*/*document-name*" – Marcus Feb 12 '16 at 9:44

I do not think there is a much more efficient way then finding files and execute a single instance of rsync. Any attampt to pipe a list of files into rsync will result in a maintained directory structure.

One alternative (if you can constraint the find pattern) is to use:

rsync -d --delete --include=*.pdf --include=*.ods --exclude=* /home/ms/archive/dir1/**/. /home/ms/newdir2/.

That will sync all pdf/ods-files of (sub-)directories with the folder /home/ms/newdir2/ without maintaining directories in a single rsync-structure.

Your given example (in case document-name or DOCUMENT-NAME) would be solved by using:

rsync -d --delete --include=*document-name.pdf --include=*DOCUMENT-NAME.pdf --include=*DOCUMENT-NAME.ods --include=*document-name.ods --include=*DOCUMENT-NAME.pdf --exclude=* /home/ms/archive/dir1/**/. /home/ms/newdir2/.
  • Is there a way to incorporate a regular expression or something for the spelling of document-name? I don't have complete control over the file naming so many of them end up DOcument-name or Document-NAME. It's impossible to guess all contingencies. Hence my use of the case insensitive flag in find. – ezgoodnight Feb 12 '16 at 16:18
  • The other problem with this is it syncs the hidden files that the find script trimmed out: -not -path '*/\.*' – ezgoodnight Feb 12 '16 at 16:24
  • Yes, I saw the flag. Unfortunately, you cannot use regex in rsync. As soon as you use find with regex, you will end up with the complete directory structure in you backup, except the case you call rsync for each found file individually (as you proposed in your question). I will keep the examples in case someone stumble upon the question and has more static filenames. – Marcus Feb 12 '16 at 16:37

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.