I've discovered that my size limits aren't necessarily in a command's length, but rather in the argument's length. Shell commands can be at 100,000-200,000 characters in length. Arguments however are limited to a few hundred characters depending on the system, as discovered by using getconf ARG_MAX
.
This leads me to the eventual problem of running out of room when I've added many folder exclusions to my rsync
arguments.
To get around this problem, I need to programmatically split up the rsync
process from one long execution, into many smaller ones.
Doing this requires "building as I go" until I hit a limit. The program will attempt to build a string of all individual exclusions, each within a list (array). If the args length limit is exceeded, it will reparse the current list to make broader exclusions, reducing the total, and store those excluded exclusions.
Here's an example of the exclusions list, as it's being built:
This list is called excludes
(long list items...)
--exclude $Home/.wine
--exclude $Home/.cache/somefolders/*
--exclude $Home/.cache/somefolders/*
--exclude $Home/.cache/somefolders/*
--exclude $Home/.cache/somefolders/*
--exclude $Home/.cache/google-chrome/Default/Cache/*
--exclude $Home/.cache/google-chrome/Default/Media\ Cache/*
**LIMIT EXCEEDED**
Now we've had our ARGS limit exceeded. The program will reparse this list looking for common folders, and create a second list to be saved for the next execution of rsync
.
After parsing, excludes
is rebuilt into a list like this:
(long list items...)
--exclude $Home/.wine
--exclude $Home/.cache/*
As it was parsed, the lines that were removed, were placed into another list, let's call it excludesnext
and looks like this:
--exclude $Home/.cache/somefolders/*
--exclude $Home/.cache/somefolders/*
--exclude $Home/.cache/somefolders/*
--exclude $Home/.cache/somefolders/*
--exclude $Home/.cache/google-chrome/Default/Cache/*
--exclude $Home/.cache/google-chrome/Default/Media\ Cache/*
So all the repeating folders are removed from excludes
and placed into excludesnext
. As items are moved, and shortened broad folders are replaced into excludes
, a new list called cutlist
is created with these new shorter folders.
So the cutlist
list includes:
--exclude $Home/.cache/*
rsync
is run using the excludes
list as the arguments.
After rsync
is finished, the excludes
list is emptied, and the excludesnext
list is copied over to the excludes
list.
Then rsync
is run in a loop using the new shortened cutlist
list as the source, and all matching subfolders from excludes
are used as exclusions.
Of course there could be many items in the cutlist
list, and the program will loop through each one. This second run of rsync
runs in a loop. But in this loop, when maximum lengths are reached, the excludes
list is parsed in the same way as before, except the new shortened folders are added to cutlist
instead of going into excludes
. This way as the loop goes through the cutlist, it can expand on itself as it goes through the list. As the excludesnext
list items are used, they are removed.
Eventually this leads to all folders and subfolders being copied over with all exclusions included regardless of how many there are. Completing the last item in cutlist
means everything has been executed.
This method can cause rsync
to run several, or even dozens of times depending how many exclusions you have, but it never repeats folders.
Apologies for not posting actual code, but it's a messy work in progress and I don't feel comfortable posting my code yet before I can release a working service.