Estimating compressed file size using a list parameter

Question

I am currently compressing a list of files from a directory in the following format:

tar -cvjf test_1.tar.gz -T test_1.lst --no-recursion

The above command will compress only those files mentioned in the list. I am doing this because this list is generated such that it fits a DVD. However, during compression the compression rate decreases the estimated file size and there is abundant space left in the DVD. This is something like a Knapsack algorithm.

I would like to estimate the compressed file size and add some more files to the list. I found that it is possible to estimate file size using the following command:

tar  -cjf - Folder/ | wc -c

This command does not take a list parameter. Is there a way to estimate compressed file size? I am also looking into options like perl scripts etc.

Answer 1

Compression will depend greatly on the repetition of the text it's compressing. If your files are plain text and all very similar, then you'll get great compression, if they are different compression will start to suffer. And if they are already compressed (i.e. JPG, MP3, etc) then compression will be even less.

As you implied with the "knapsack" algorithm, with sufficient time and CPU resources, it should be possible to try all combinations of files in a single compressed tar file. Unfortunately that could take way too long.

Assuming your files are similar (i.e all are mostly text, or all are similar files), then you might be able to get a close estimate by compressing each file individually and then adding the sizes together for an estimate at a total compress tar archive.

The only downfall is the estimate will fail and produce a smaller final tar.bz2 file if the files are similar and the compression algorithm can use the similarities to it's advantage.