2

I'm trying to run a backup of all my .x folders in Home to save configurations.

With the program I'm writing, you can select any folders you want included or excluded from the backup, this list is dynamic.

I'm trying to build something like this:

rsync -aP --exclude $Home/.cache/google-chrome/Default/Cache/* --exclude $Home/.cache/google-chrome/Default/Media\ Cache/* --exclude $Home/.wine $HOME/.* /mnt/ext/

So in the above example, everything is copied to an external drive except for Wine and Chrome's caches. I'm aware that I can built a text file and tell rsync --exclude-from 'textfile' but I'd rather not have to read and rewrite a text file.

My worry is that as this expands, the exclusion list can get quite long, since my project will eventually start including folders outside of $HOME, and on other systems' NFS shares. I can launch the command and use the excludes list in a variable:

exlist = "--exclude $Home/.cache/google-chrome/Default/Cache/* --exclude $Home/.cache/google-chrome/Default/Media\ Cache/* --exclude $Home/.wine" etc etc
subprocess.Popen("rsync -options ",exlist," /source /dest")

But is this going to cause problems with extremely long shell commands where I'll have to break it up into smaller chunks? I'd rather stay ahead of potential future problems and begin writing handlers to accomplish this now rather than try to fix and patch it later.

0

2 Answers 2

1

I've discovered that my size limits aren't necessarily in a command's length, but rather in the argument's length. Shell commands can be at 100,000-200,000 characters in length. Arguments however are limited to a few hundred characters depending on the system, as discovered by using getconf ARG_MAX.

This leads me to the eventual problem of running out of room when I've added many folder exclusions to my rsync arguments.

To get around this problem, I need to programmatically split up the rsync process from one long execution, into many smaller ones.

Doing this requires "building as I go" until I hit a limit. The program will attempt to build a string of all individual exclusions, each within a list (array). If the args length limit is exceeded, it will reparse the current list to make broader exclusions, reducing the total, and store those excluded exclusions.

Here's an example of the exclusions list, as it's being built:

This list is called excludes

(long list items...)  
--exclude $Home/.wine  
--exclude $Home/.cache/somefolders/*   
--exclude $Home/.cache/somefolders/*  
--exclude $Home/.cache/somefolders/*  
--exclude $Home/.cache/somefolders/*   
--exclude $Home/.cache/google-chrome/Default/Cache/*  
--exclude $Home/.cache/google-chrome/Default/Media\ Cache/* 
**LIMIT EXCEEDED**

Now we've had our ARGS limit exceeded. The program will reparse this list looking for common folders, and create a second list to be saved for the next execution of rsync.

After parsing, excludes is rebuilt into a list like this:

(long list items...)  
--exclude $Home/.wine  
--exclude $Home/.cache/*  

As it was parsed, the lines that were removed, were placed into another list, let's call it excludesnext and looks like this:

--exclude $Home/.cache/somefolders/*   
--exclude $Home/.cache/somefolders/*  
--exclude $Home/.cache/somefolders/*  
--exclude $Home/.cache/somefolders/*   
--exclude $Home/.cache/google-chrome/Default/Cache/*  
--exclude $Home/.cache/google-chrome/Default/Media\ Cache/*

So all the repeating folders are removed from excludes and placed into excludesnext. As items are moved, and shortened broad folders are replaced into excludes, a new list called cutlist is created with these new shorter folders.

So the cutlist list includes:

--exclude $Home/.cache/* 

rsync is run using the excludes list as the arguments.

After rsync is finished, the excludes list is emptied, and the excludesnext list is copied over to the excludes list.

Then rsync is run in a loop using the new shortened cutlist list as the source, and all matching subfolders from excludes are used as exclusions.

Of course there could be many items in the cutlist list, and the program will loop through each one. This second run of rsync runs in a loop. But in this loop, when maximum lengths are reached, the excludes list is parsed in the same way as before, except the new shortened folders are added to cutlist instead of going into excludes. This way as the loop goes through the cutlist, it can expand on itself as it goes through the list. As the excludesnext list items are used, they are removed.

Eventually this leads to all folders and subfolders being copied over with all exclusions included regardless of how many there are. Completing the last item in cutlist means everything has been executed.

This method can cause rsync to run several, or even dozens of times depending how many exclusions you have, but it never repeats folders.

Apologies for not posting actual code, but it's a messy work in progress and I don't feel comfortable posting my code yet before I can release a working service.

0

I'm aware that I can built a text file and tell rsync --exclude-from 'textfile' but I'd rather not have to read and rewrite a text file.

Just do:

with contextlib.closing(tempfile.NamedTemporaryFile()) as exclude_from:
    print(*your_exclude_list, sep="\n", flush=True, file=exclude_from)  # etc
    subprocess.check_call(['rsync', '--exclude-from', exclude_from.name, ...])

...and don't worry about the temporary file. I appreciate that temporary files seem messy to deal with, but with Python's library and context managers this can all be nicely wrapped in a parcel with a bow so you don't have to worry about it.

4
  • This should work as well, however, the system will be full on read-only, and since it will receive its instruction over the network via tcp, I don't want anything written to the local disc. Backups will have source and destinations over NFS system. I'm hoping to minimize this system into something as small as possible, maybe a RaspberryPi or smaller.
    – Delorean
    Jun 28, 2016 at 15:22
  • Just mount a tmpfs. Something like mkdir -p /tmp/tmpfs; mount -t tmpfs none /tmp/tmpfs; export TMPDIR=/tmp/tmpfs. Or given that your entire system is read-only, you probably want /tmp itself to be a tmpfs. You can adjust /etc/fstab to make that happen automatically. Stuff generally expects a working temporary directory. If you don't have one, your Python program may be fine but other things may break. So you're best off doing this anyway, whether you use this solution or not. Jun 29, 2016 at 11:25
  • Thanks for that workaround, and I'll probably use it in another build. But this one like I said, is locked down and very minimal. The entire OS will run off of a flash drive which I why I don't want any writes at all. Networking will be very restricted, and nothing else will be running other than my python program, rsync, and ssh server for me to log in, so I'm not concerned about any programs not being able to write anywhere and I'll be doing everything in memory.
    – Delorean
    Jun 29, 2016 at 14:29
  • tmpfs is a memory filesystem - it doesn't involve any writes to disk at all (unless you're using swap, which presumably you aren't). Jun 29, 2016 at 17:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .