Totally patchwork and a quick, rough sketch as it is, but tested on a directory with 3000 files, the script below did an extremely fast job:
#!/usr/bin/env python3
import subprocess
import os
import sys
splitinto = 2
dr = sys.argv[1]
os.chdir(dr)
files = os.listdir(dr)
n_files = len(files)
size = n_files // splitinto
def compress(tar, files):
command = ["tar", "-zcvf", "tarfile" + str(tar) + ".tar.gz", "-T", "-", "--null"]
proc = subprocess.Popen(command, stdin=subprocess.PIPE)
with proc:
proc.stdin.write(b'\0'.join(map(str.encode, files)))
proc.stdin.write(b'\0')
if proc.returncode:
sys.exit(proc.returncode)
sub = []; tar = 1
for f in files:
sub.append(f)
if len(sub) == size:
compress(tar, sub)
sub = []; tar += 1
if sub:
# taking care of left
compress(tar, sub)
How to use
numbered .tar.gz
files will be created in the same directory as where the files are.
Explanation
The script:
- lists all files in the directory
- cd's into the directory to prevent adding the path info to the tar file
- reads through the file list, grouping them by the set division
- compresses the sub group(s) into numbered files
EDIT
Automatically create chunks by size in mb
More sophisticated is to use the max- size (in mb) of the chunks as a (second) argument. In the script below, the chunks are written into a compressed file as soon as the chunk reaches (passes) the threshold.
Since the script is triggered by the chunks, exceeding the threshold, this will only work if the size of (all) files is substantially smaller than the chunk size.
The script:
#!/usr/bin/env python3
import subprocess
import os
import sys
dr = sys.argv[1]
chunksize = float(sys.argv[2])
os.chdir(dr)
files = os.listdir(dr)
n_files = len(files)
def compress(tar, files):
command = ["tar", "-zcvf", "tarfile" + str(tar) + ".tar.gz", "-T", "-", "--null"]
proc = subprocess.Popen(command, stdin=subprocess.PIPE)
with proc:
proc.stdin.write(b'\0'.join(map(str.encode, files)))
proc.stdin.write(b'\0')
if proc.returncode:
sys.exit(proc.returncode)
sub = []; tar = 1; subsize = 0
for f in files:
sub.append(f)
subsize = subsize + (os.path.getsize(f)/1000000)
if subsize >= chunksize:
compress(tar, sub)
sub = []; tar += 1; subsize = 0
if sub:
# taking care of left
compress(tar, sub)
To run:
python3 /path/tocompress_split.py /directory/with/files/tocompress chunksize
...where chunksize is the size of input for the tar command.
In this one, the suggested improvements by @DavidFoerster are included. Thanks a lot!
tar
them by adding all the files starting with a certain pattern until you have them all. This can be easily scripted but does not guarantee the size will be lower than 9MB as you need. You could, however, manually adjust the size of those files that are too large by splitting them further.