I have a directory with many subdirectories and several million files. Total size around 15GB. I have a brief window (preferably around 30 seconds) to mirror changes to this directory to a second identical directory on another drive. Drives are SSD.

I have tried various incarnations of tar and rsync, but landed on "cp -ru" because it is able to copy the new and changed files within the time allowed. The first cp takes maybe 15 minutes; but future "cp -ru" only take 20 to 40 seconds. Other options I tried were far slower (taking many minutes).

My problem is I end up with "extra" files. Files that may have briefly existed on source end up on destination forever; even after they were deleted from source. Is there a way to modify the "cp -ru" to eliminate destination files that no longer exist in source? Or would that slow it down too much?

Or maybe a secondary operation could somehow scan through the source in a "read only" way and eliminate files in the destination? This operation could take longer; as I have about a 1 hour time period between backups. I just can't actually copy files during this period because they are on a live server (files are being modified).

I'm willing to try other options than cp; but it needs to be able to handle backing up changes to millions of files within around 30 seconds.

I am running Ubuntu Server 14.04.3 LTS.

  • 4
    For that you could use rsync --delete. – muru Jan 28 '16 at 17:43
  • Would you give full command please? I'm not that familiar with rsync. Will it be fast enough? – Mark Krueger Jan 28 '16 at 21:09

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.