2

I have a folder with 7,491 files in it. I'd like to tar gzip these in batches of 100, e.g.

first 100 files > archive1.tar.gz
files 101-200 > archive2.tar.gz
files 201-300 > archive3.tar.gz

Is there a way to do this via the command line?

7

The simplest option might be to use the split command in combination with find and tar's -T/--files-from option:

find ... -type f -print0 | 
  split -t '\0' -l 100 -d --filter 'tar czf archive-$FILE.tar.gz --files-from=- --null'

split will also take care of setting filenames with sequential numbering (the -d option sets numbered filenames instead of the usual alphabetic ones).

With a combination of find and xargs, I couldn't find a way to get reliable sequential numbering of the archives, so I used uuidgen:

find <directory> -type f -print0 | xargs -n 100 -0 sh -c 'tar -czf archive-$(uuidgen).tar.gz "$@"' _

Or you could use bash loops:

count=1
N=100
find <directory> -type f -print0  | 
{
while true
do
    files=()
    for ((i=0;i<N;i++))   # Add N filenames to the `files` array
    do
        IFS= read -d '' -r file
        files+=("$file")
    done
    ((count++))
    tar -czf archive-$count.tar.gz "${files[@]}"
done
}

An assumption in all these commands is that find is not being run in the current directory. Otherwise, you might end up tarring a tar file. If you have to run it in the current directory, then I suggest saving the filenames to a file and then reading the filenames from that file for split/xargs/the loop.

4
  • What if each batch has many many files (say, 100,000)? – Stan Jan 2 '18 at 11:38
  • 1
    @Stan the -T/--files-from option would be best in that case. Maybe something like find ... -print0 | split -t '\0' -l 100000 --filter 'tar czf archive-$FILE.tar.gz --files-from=- --null' – muru Jan 2 '18 at 12:04
  • Verified. Good solution! – Stan Jan 3 '18 at 0:55
  • For anyone who finds split lacks the -t option, may try find target_path -type f | split -l file_number_per_package -d --filter 'tar cvf PackageName-$FILE.tar --files-from=- --null' – Stan Apr 4 '18 at 8:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.