10

I'm getting started with rsync and tried to use it to keep two folders on local system synced. I have a source folder, whose content changes over time (some files get added, some changes and some deleted) and a target folder that I want to almost be a mirror of the source. So what I tried was using rsync like this:

rsync -a --delete "${source_dir}" "${target_dir}";

This does keep the contents of target the exact same as the contents of source. However, I would like to be able to add some files to target and not to source, but I don't want them to be deleted everytime I do rsync. On the other hand, files that used to be synced and then got deleted in source should still be deleted.

Is there a way to do this without having to alter the command for every file that I want excluded?

Update: I should mention that I am not limited to rsync. If another program gets the job done, that's fine too. I just tried to solve this using rsync.

  • Hi @AszunesHeart, just curious, but did you test the answer(s)? – Jacob Vlijm Feb 4 '17 at 6:23
  • Have you tried taking out the --delete option? That one is like the /MIR option in Robocopy. – SDsolar Jul 26 '17 at 8:58
9

rsync has an option called --exclude-from option which allows you to create a file containing a list of any files which you would like to exclude. You can update this file whenever you want to add a new exclusion, or remove an old one.

If you create the exclude file at /home/user/rsync_exclude the new command would be:

rsync -a --delete --exclude-from="/home/user/rsync_exclude" "${source_dir}" "${target_dir}"

When creating the exclude list file, you should put each exclusion rule on a separate line. The exclusions are relative to your source directory. If your /home/user/rsync_exclude file contained the following options:

secret_file
first_dir/subdir/*
second_dir/common_name.*
  • Any file or directory called secret_file in your source directory will be excluded.
  • Any files in ${source_dir}/first_dir/subdir will be excluded, but an empty version of subdir will be synced.
  • Any files in ${source_dir}/second_dir with a prefix of common_name. will be ignored. So common_name.txt, common_name.jpg etc.
  • I'm not sure if this does what I wanted. Also I find it impractical to list every file or folder that gets added to target. I'd rather like to have an automatic way to do that. Let's say I have various scripts in target that produce multiple log-files (also in target) and I don't want to list every location of those files in the rsync_exclude-file. Is there a way to make rsync "remember" which files where synced and only let those be affected by --delete? – jkrzefski Sep 1 '16 at 8:40
  • Sorry, I misread your question, I though you wanted to add to source, and have those not update to target. I think there's a way to do what you want, but I'll have to ponder it for a bit. I'll comment once I've had time to edit. – Arronical Sep 1 '16 at 8:43
  • @jkrzefski If you are producing files from another script in target, and want to exclude them from the source, then why not to change the destination of those log files to another folder? Presumably, if you are not syncing them, is because they are less important. – user308164 Jan 31 '17 at 11:27
6
+25

Since you mentioned: I am not limited to rsync:

Script to maintain the mirror, allowing to add extra files to target

Below a script that does exactly what you describe.

The script can be run in verbose mode (to be set in the script), which will output the progress of the backup (mirroring). No need to say this can also be used to log the backups:

Verbose option

enter image description here


The concept

1. On first backup, the script:

  • creates a file (in the target directory), where all files and directories are listed; .recentfiles
  • creates an exact copy (mirror) of all files and directories in the target directory

2. On the next and so on backup

  • The script compares directory structure and modification date(s) of the files. New files and dirs in the source are copied to the mirror. At the same time a second (temporary) file is created, listing the current files and dirs in the source directory; .currentfiles.
  • Subsequently, .recentfiles (listing the situation on previous backup) is compared to .currentfiles. Only files from .recentfiles which are not in .currentfiles are obviously removed from the source, and will be removed from the target.
  • Files you manually added to the target folder are not in anyway "seen" by the script, and are left alone.
  • Finally, the temporary .currentfiles is renamed to .recentfiles to serve the next backup cycle and so on.

The script

#!/usr/bin/env python3
import os
import sys
import shutil

dr1 = sys.argv[1]; dr2 = sys.argv[2]

# --- choose verbose (or not)
verbose = True
# ---

recentfiles = os.path.join(dr2, ".recentfiles")
currentfiles = os.path.join(dr2, ".currentfiles")

if verbose:
    print("Counting items in source...")
    file_count = sum([len(files)+len(d) for r, d, files in os.walk(dr1)])
    print(file_count, "items in source")
    print("Reading directory & file structure...")
    done = 0; chunk = int(file_count/5); full = chunk*5

def show_percentage(done):
    if done % chunk == 0:
        print(str(int(done/full*100))+"%...", end = " ")

for root, dirs, files in os.walk(dr1):
    for dr in dirs:
        if verbose:
            if done == 0:
                print("Updating mirror...")
            done = done + 1
            show_percentage(done) 
        target = os.path.join(root, dr).replace(dr1, dr2)
        source = os.path.join(root, dr)
        open(currentfiles, "a+").write(target+"\n")
        if not os.path.exists(target):
            shutil.copytree(source, target)
    for f in files:
        if verbose:
            done = done + 1
            show_percentage(done)
        target = os.path.join(root, f).replace(dr1, dr2)
        source = os.path.join(root, f)
        open(currentfiles, "a+").write(target+"\n") 
        sourcedit = os.path.getmtime(source)
        try:
            if os.path.getmtime(source) > os.path.getmtime(target):
                shutil.copy(source, target)   
        except FileNotFoundError:
            shutil.copy(source, target)

if verbose:
    print("\nChecking for deleted files in source...")

if os.path.exists(recentfiles):
    recent = [f.strip() for f in open(recentfiles).readlines()]
    current = [f.strip() for f in open(currentfiles).readlines()]
    remove = set([f for f in recent if not f in current])
    for f in remove:
        try:
            os.remove(f)
        except IsADirectoryError:
            shutil.rmtree(f)
        except FileNotFoundError:     
            pass
        if verbose:
            print("Removed:", f.split("/")[-1])

if verbose:
    print("Done.")

shutil.move(currentfiles, recentfiles)

How to use

  1. Copy the script into an empty file, save it as backup_special.py
  2. Change -if you want- the verbose option in the head of the script:

    # --- choose verbose (or not)
    verbose = True
    # ---
    
  3. Run it with source and target as arguments:

     python3 /path/to/backup_special.py <source_directory> <target_directory>
    

Speed

I tested the script on a 10 GB directory with some 40.000 files and dirs on my network drive (NAS), it made the backup in pretty much the same time as rsync.

Updating the whole directory took only a few seconds more than rsync, on 40.000 files, which is imo acceptable and no surprise, since the script needs to compare the content to the last made backup.

  • Hi @Aszune'sHeart added a scripted option. Please mention if all is clear. – Jacob Vlijm Jan 31 '17 at 11:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.