I have a directory with many subdirectories and several million files. Total size around 15GB. I have a brief window (preferably around 30 seconds) to mirror changes to this directory to a second identical directory on another drive. Drives are SSD.
I have tried various incarnations of tar and rsync, but landed on "cp -ru" because it is able to copy the new and changed files within the time allowed. The first cp takes maybe 15 minutes; but future "cp -ru" only take 20 to 40 seconds. Other options I tried were far slower (taking many minutes).
My problem is I end up with "extra" files. Files that may have briefly existed on source end up on destination forever; even after they were deleted from source. Is there a way to modify the "cp -ru" to eliminate destination files that no longer exist in source? Or would that slow it down too much?
Or maybe a secondary operation could somehow scan through the source in a "read only" way and eliminate files in the destination? This operation could take longer; as I have about a 1 hour time period between backups. I just can't actually copy files during this period because they are on a live server (files are being modified).
I'm willing to try other options than cp; but it needs to be able to handle backing up changes to millions of files within around 30 seconds.
I am running Ubuntu Server 14.04.3 LTS.
rsync --delete.