11

I'm on Ubuntu 16.04.

I have a folder with a lot of text files (almost 12k). I need to upload them all to a website that accepts .tar.gz uploads and then decompresses them automatically, but has a limit of 10MB (10000KB) per file (so in particular each file has to be decompressed on its own). If I tar.gz all these files the resulting file is of about 72MB.

What I would like to do is to create eight .tar.gz files, each of size / dimension (strictly) smaller than 10000KB.

Alternatively, one can assume that all the files above have approximately the same dimension, so I would like to create eight .tar.gz files with more or less the same amount of files each.

How can I do any of these two tasks?

I am perfectly fine with a solution that involves GUI, CLI or scripting. I am not looking for speed here, I just need it done.

1
  • Presumably the 12k files you have will have patterns or repeated characters in their names. You could possibly tar them by adding all the files starting with a certain pattern until you have them all. This can be easily scripted but does not guarantee the size will be lower than 9MB as you need. You could, however, manually adjust the size of those files that are too large by splitting them further. Nov 6 '16 at 12:35
9

Totally patchwork and a quick, rough sketch as it is, but tested on a directory with 3000 files, the script below did an extremely fast job:

#!/usr/bin/env python3
import subprocess
import os
import sys

splitinto = 2

dr = sys.argv[1]
os.chdir(dr)

files = os.listdir(dr)
n_files = len(files)
size = n_files // splitinto

def compress(tar, files):
    command = ["tar", "-zcvf", "tarfile" + str(tar) + ".tar.gz", "-T", "-", "--null"]
    proc = subprocess.Popen(command, stdin=subprocess.PIPE)
    with proc:
        proc.stdin.write(b'\0'.join(map(str.encode, files)))
        proc.stdin.write(b'\0')
    if proc.returncode:
        sys.exit(proc.returncode)

sub = []; tar = 1
for f in files:
    sub.append(f)
    if len(sub) == size:
        compress(tar, sub)
        sub = []; tar += 1

if sub:
    # taking care of left
    compress(tar, sub)

How to use

  • Save it into an empty file as compress_split.py
  • In the head section, set the number of files to compress into. In practice, there will always be one more to take care of the remaining few "left overs".
  • Run it with the directory with your files as argument:

    python3 /path/tocompress_split.py /directory/with/files/tocompress
    

numbered .tar.gz files will be created in the same directory as where the files are.

Explanation

The script:

  • lists all files in the directory
  • cd's into the directory to prevent adding the path info to the tar file
  • reads through the file list, grouping them by the set division
  • compresses the sub group(s) into numbered files

EDIT

Automatically create chunks by size in mb

More sophisticated is to use the max- size (in mb) of the chunks as a (second) argument. In the script below, the chunks are written into a compressed file as soon as the chunk reaches (passes) the threshold.

Since the script is triggered by the chunks, exceeding the threshold, this will only work if the size of (all) files is substantially smaller than the chunk size.

The script:

#!/usr/bin/env python3
import subprocess
import os
import sys

dr = sys.argv[1]
chunksize = float(sys.argv[2])
os.chdir(dr)

files = os.listdir(dr)
n_files = len(files)

def compress(tar, files):
    command = ["tar", "-zcvf", "tarfile" + str(tar) + ".tar.gz", "-T", "-", "--null"]
    proc = subprocess.Popen(command, stdin=subprocess.PIPE)
    with proc:
        proc.stdin.write(b'\0'.join(map(str.encode, files)))
        proc.stdin.write(b'\0')
    if proc.returncode:
        sys.exit(proc.returncode)

sub = []; tar = 1; subsize = 0
for f in files:
    sub.append(f)
    subsize = subsize + (os.path.getsize(f)/1000000)
    if subsize >= chunksize:
        compress(tar, sub)
        sub = []; tar += 1; subsize = 0

if sub:
    # taking care of left
    compress(tar, sub)

To run:

python3 /path/tocompress_split.py /directory/with/files/tocompress chunksize

...where chunksize is the size of input for the tar command.

In this one, the suggested improvements by @DavidFoerster are included. Thanks a lot!

7
  • @dadexix86 you're welcome! Nov 6 '16 at 14:05
  • I god rid of the shell invocation and used an argument list directly. Still, large argument lists may be problematic and I'll try to improve the tar invocation further by providing the file list on the standard input stream. Nov 6 '16 at 14:18
  • Hi @DavidFoerster , I trust your insight, but what is the advantage? Nov 6 '16 at 14:52
  • Most runtime environments have a (soft and hard) limit on the total length of the argument strings of a command which you'll reach quickly when operating on thousands of files. That's why tar allows you to specify files to add (or extract) on standard input with an appropriate option. Nov 6 '16 at 14:58
  • @DavidFoerster there is an issue though, the second one does not run anymore. Actually neither of them does... Nov 6 '16 at 15:02
6

A pure shell approach:

files=(*); 
num=$((${#files[@]}/8));
k=1
for ((i=0; i<${#files[@]}; i+=$num)); do 
    tar cvzf files$k.tgz -- "${files[@]:$i:$num}"
    ((k++))
done

Explanation

  • files=(*) : save the list of files (also directories if any are present, change to files=(*.txt) to get only things with a txt extension) in the array $files.
  • num=$((${#files[@]}/8)); : ${#files[@]} is the number of elements in the array $files. The $(( )) is bash's (limited) way of doing arithmetic. So, this command sets $num to the number of files divided by 8.
  • k=1 : just a counter to name the tarballs.
  • for ((i=0; i<${#files[@]}; i+=$num)); do : iterate over the values of the array. $i is initialized at 0 (the first element of the array) and incremented by $num. This continues until we've gone through all elements (files).
  • tar cvzf files$i.tgz -- ${files[@]:$i:$num} : in bash, you can get an array slice (part of an array) using ${array[@]:start:length}, So ${array[@]:2:3} will return three elements starting from the second. Here, we are taking a slice that starts at the current value of $i and is $num elements long. The -- is needed in case any of your file names can start with a -.
  • ((k++)) : increment $k
2
  • Nice! First time I've seen a practical use of bash array index ranges.
    – Joe
    Nov 10 '16 at 5:53
  • Very clean and succinct. To me, more understandable than the Python solutions though both are pretty good. Wonder how they all compare in performance? Nov 10 '16 at 8:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.