2

I have a folder with 7,491 files in it. I'd like to tar gzip these in batches of 100, e.g.

first 100 files > archive1.tar.gz
files 101-200 > archive2.tar.gz
files 201-300 > archive3.tar.gz

Is there a way to do this via the command line?

1 Answer 1

7

The simplest option might be to use the split command in combination with find and tar's -T/--files-from option:

find ... -type f -print0 | 
  split -t '\0' -l 100 -d --filter 'tar czf archive-$FILE.tar.gz --files-from=- --null'

split will also take care of setting filenames with sequential numbering (the -d option sets numbered filenames instead of the usual alphabetic ones).

With a combination of find and xargs, I couldn't find a way to get reliable sequential numbering of the archives, so I used uuidgen:

find <directory> -type f -print0 | xargs -n 100 -0 sh -c 'tar -czf archive-$(uuidgen).tar.gz "$@"' _

Or you could use bash loops:

count=1
N=100
find <directory> -type f -print0  | 
{
while true
do
    files=()
    for ((i=0;i<N;i++))   # Add N filenames to the `files` array
    do
        IFS= read -d '' -r file
        files+=("$file")
    done
    ((count++))
    tar -czf archive-$count.tar.gz "${files[@]}"
done
}

An assumption in all these commands is that find is not being run in the current directory. Otherwise, you might end up tarring a tar file. If you have to run it in the current directory, then I suggest saving the filenames to a file and then reading the filenames from that file for split/xargs/the loop.

4
  • What if each batch has many many files (say, 100,000)?
    – Stan
    Jan 2, 2018 at 11:38
  • 1
    @Stan the -T/--files-from option would be best in that case. Maybe something like find ... -print0 | split -t '\0' -l 100000 --filter 'tar czf archive-$FILE.tar.gz --files-from=- --null'
    – muru
    Jan 2, 2018 at 12:04
  • Verified. Good solution!
    – Stan
    Jan 3, 2018 at 0:55
  • For anyone who finds split lacks the -t option, may try find target_path -type f | split -l file_number_per_package -d --filter 'tar cvf PackageName-$FILE.tar --files-from=- --null'
    – Stan
    Apr 4, 2018 at 8:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .