74

What compression tools are available in Ubuntu that can benefit from a multi-core CPU.

Improve this question
2
  • Just for the record, an alternative may be to create independent archives in parallel. So instead of creating myfiles.8core.xz, you create myfiles1.xz to myfiles8.xz in parallel. This will require a dispatch agent. Both approaches have complementary pros and cons.
    – Asclepius
    Apr 17, 2014 at 21:05
  • 2
    Tried to unzip a 7GB file using bzip2 only to find out it is not using all my 8 cores. Read about it and decided to try pbzip2. Still running on just one core. Then I noticed comments saying pbzip2 can only fully parallelize decompression of files it compressed itself. Same comments suggested lbzip2 can fully parallelize on any bz2 file which indeed was true - made almost full use (80-90% of the CPU) of all my cores and it decompressed way faster.
    – Edi Bice
    Sep 8, 2014 at 15:17

9 Answers 9

Reset to default
84

Well, the keyword was parallel. After looking for all compression tools that were also parallel I found the following:

PXZ - Parallel XZ is a compression utility that takes advantage of running LZMA compression of different parts of an input file on multiple cores and processors simultaneously. Its primary goal is to utilize all resources to speed up compression time with minimal possible influence on compression ratio.

sudo apt-get install pxz

PLZIP - Lzip is a lossless data compressor based on the LZMA algorithm, with very safe integrity checking and a user interface similar to the one of gzip or bzip2. Lzip decompresses almost as fast as gzip and compresses better than bzip2, which makes it well suited for software distribution and data archiving.

Plzip is a massively parallel (multi-threaded) version of lzip using the lzip file format; the files produced by plzip are fully compatible with lzip.

Plzip is intended for faster compression/decompression of big files on multiprocessor machines, which makes it specially well suited for distribution of big software files and large scale data archiving. On files big enough, plzip can use hundreds of processors.

sudo apt-get install plzip

PIGZ - pigz, which stands for Parallel Implementation of GZip, is a fully functional replacement for gzip that takes advantage of multiple processors and multiple cores when compressing data.

sudo apt-get install pigz

PBZIP2 - pbzip2 is a parallel implementation of the bzip2 block-sorting file compressor that uses pthreads and achieves near-linear speedup on SMP machines. The output of this version is fully compatible with bzip2 v1.0.2 (ie: anything compressed with pbzip2 can be decompressed with bzip2).

sudo apt-get install pbzip2

LRZIP - A multithreaded compression program that can achieve very high compression ratios and speed when used with large files. It uses the combined compression algorithms of zpaq and lzma for maximum compression, lzo for maximum speed, and the long range redundancy reduction of rzip. It is designed to scale with increases with RAM size, improving compression further. A choice of either size or speed optimizations allows for either better compression than even lzma can provide, or better speed than gzip, but with bzip2 sized compression levels.

sudo apt-get install lrzip

A small Compression Benchmark (Using the test Oli created):

ORIGINAL FILE SIZE - 100 MB
PBZIP2 - 101 MB (1% Bigger)
PXZ - 101 MB (1% Bigger)
PLZIP - 102 MB (1% Bigger)
LRZIP - 101 MB (1% Bigger)
PIGZ - 101 MB (1% Bigger)

A small Compression Benchmark (Using a Text file):

ORIGINAL FILE SIZE - 70 KB Text File
PBZIP2 - 16.1 KB (23%)
PXZ - 15.4 KB (22%)
PLZIP - 15.5 KB (22.1%)
LRZIP - 15.3 KB (21.8%)
PIGZ - 17.4 KB (24.8%)

Improve this answer
5
  • Examples would be great.
    – earthmeLon
    Nov 4, 2014 at 21:38
  • @earthmeLon Read the answer by Oli which mentions how to create the example file. Then proceed with the commands I used.
    – Luis Alvarado
    Nov 4, 2014 at 22:09
  • I hope the output of these are inter-compatible. i.e. output from lrzip can be uncompressed using pbzip2, for instance.
    – Vineet Menon
    Sep 24, 2018 at 5:16
  • Great list and thanks! In addition, unless you are searching for general set of replacements, I would make sure to test your usecase. The speeds and compression ratios are going to depend a bit on what you are compressing. I was just using bzip2 and have just tested all the options above. pbzip2 compresses 4x faster for <1% bigger. The best compression was lzma, but it took 8x std bzip2, xz was about the same as std bzip2 for time with about 6% better compression. pigz gave the crappy compression you'd expect, but was 32x faster than std bzip2 (1sec vs 32secs)!!!
    – AntonOfTheWoods
    Nov 10, 2020 at 14:35
  • Cool, now I am using pxz instead of gz and get a additional decrease in size of the compressed file of nearly 50 %. I get my mysqldump-data, pxz and an upload to an other server in the same time. (170 MB Data => 23 MB.gz, now 11 MB .xz)
    – Martin T.
    Feb 5, 2021 at 12:57
36

There are two main tools. lbzip2 and pbzip2. They're essentially different implementations of bzip2 compressors. I've compared them (the output is a tidied up version but you should be able to run the commands)

cd /dev/shm  # we do all of this in RAM!
dd if=/dev/urandom of=bigfile bs=1024 count=102400

$ lbzip2 -zk bigfile 
Time: 0m3.596s
Size: 105335428 

$ pbzip2 -zk bigfile
Time: 0m5.738s6
Size: 10532460

lbzip2 appears to be the winner on random data. It's slightly less compressed but much quicker. YMMV.

Improve this answer
8
  • 5
    looks like a digit is missing from the pbzip2 Size
    – Wayne Walker
    Sep 6, 2014 at 22:35
  • 6
    /dev/urandom is not a great choice of input for benchmarking compression tools, since random data is, by definition, incompressible. That partly explains why in both cases the output file is ~450MiB bigger than the input.
    – ali_m
    Mar 2, 2016 at 14:22
  • 1
    Sorry, I'm being really pedantic but truly random data can be super-compressible. You could ask a perfect RNG for 32 bits and get 00000000000000000000000000000000. That's how random works ;) What you're talking about is practical averages. It's unlikely you'll generate a 100MB file of just zeros. And I agree with the spirit of what you're saying, I just don't agree with the "by definition" because that isn't the definition (because it's inaccurate).
    – Oli ♦
    Mar 2, 2016 at 15:22
  • 2
    When we're judging the performance of different compression methods, what we're really interested in is the expected output size for future examples of the kind of data we want to compress. If this data is truly random then it contains no statistical regularity for compression to exploit, so for sequences of N random bytes the best we could ever hope for is an expected output length of N bytes. For some examples we might do a bit better, for others we might do a bit worse (in practice we almost always do worse), but the expected output length stays the same.
    – ali_m
    Mar 2, 2016 at 15:35
  • 6
    I mean "random" in the Kolmogorov sense, which is literally defined as incompressibility. There's no universal benchmark for compression since different algorithms work better for different types of data. A good start might be just to pipe it some text, e.g. wget http://mattmahoney.net/dc/enwik8.zip to grab 96MB (21MB compressed) of text from Wikipedia. For a much more comprehensive suite of benchmarks, see here.
    – ali_m
    Mar 2, 2016 at 16:36
24

Update:

XZ Utils supports multi-threaded compression since v5.2.0, it was originally mistakenly documented as being multi-threaded decompression.

For example: tar -cf - source | xz --threads=0 > destination.tar.xz

Improve this answer
4
  • 2
    You can also run export XZ_DEFAULTS="-T 0" and then just use your usual tar call, i.e. tar cJf target.tar.xz source.
    – scai
    Dec 28, 2018 at 12:29
  • 1
    wow! why is this not the default?
    – gimix
    Jun 10, 2020 at 7:43
  • tar -cvf ... to be verbose
    – Stuart Cardall
    Jul 9, 2020 at 18:48
  • on my Linux xz --threads=0 is actually utilizing 3/8 threads, so not really parallel, pixz or as some people said pxz is utilizing 5/8, so I would suggest using pxz for better results.
    – Georgi Stoyanov
    Nov 10, 2020 at 9:06
11

In addition the nice summary above (thanks Luis), these days folks might also want to consider PIXZ, which according to it's README (Source: https://github.com/vasi/pixz -- I haven't verified the claims myself) has some advantages over PXZ.

[Compared to PIXZ, PXZ has these advantages and disadvantages:]

    * Simpler code
    * Uses OpenMP instead of pthreads
    * Uses streams instead of blocks, not indexable
    * Uses temp files and doesn't combine them until the whole file is compressed, high disk/memory usage

In other words, PIXZ is supposedly more memory and disk efficient, and has an optional indexing feature that speeds up decompression of individual components of compressed tar files.

Improve this answer
3
  • However, it is my understanding that pixz archives are not compatible with standard xz format, the way pxz would be.
    – Mxx
    Jun 30, 2014 at 18:04
  • 5
    @Mxx: The file formats are compatible. pixz can decompress xz archives and xz can decompress pixz archives. However, the command line options on xz and pixz differ.
    – Snowball
    Oct 24, 2014 at 7:24
  • Indexable files are a big win for pixz.
    – ostrokach
    Feb 2, 2016 at 9:37
7

Zstandard supports multi-threading since v1.2.0¹. It is a very fast compressor and decompressor intended to replace gzip and it can also compress as efficient (if not better) as LZMA2/XZ on its highest levels.

You have to use one of these releases, or compile the latest version from source to get these benefits. Luckily it doesn't pull in a lot of dependencies.

There was also a 3rd party pzstd in v1.1.0 of zstd.

Improve this answer
5

lzop may also be a viable option, although it's single-threaded.

It uses the very fast lempel-ziv-oberhumer compression algorithm which is 5-6 times faster than gzip in my observation.

Note: Although it's not multi-threaded yet, it will probably outperform pigz on 1-4 core systems. That's why I decided to post this even if it doesn't directly answer your question. Try it, it may solve your CPU bottleneck problem while using only one CPU and compressing a little worse. I found it often to be a better solution than, e.g pigz.

Improve this answer
3
  • Isn't it just better at decompressing? Compressing takes about the same (or worse) than gzip
    – Mr. Developerdude
    May 28, 2014 at 23:36
  • 1
    I can also testify that lzop is super fast. Proxmox uses lzop for backup of virtual machines by default.
    – Lonnie Best
    Oct 14, 2014 at 17:34
  • 2
    lz4 is even faster (and has a multi-threaded version).
    – David Balažic
    Jul 25, 2016 at 12:57
4

It is not really an answer, but I think it is relevant enough to share my benchmarks comparing speed of gzip and pigz on a real HW in a real life scenario. As pigz is the multithreaded evolution I personally have chosen to use from now on.

Metadata:

  • Hardware used: Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz (4c/8t) + Nvme SSD
  • GNU/Linux distribution: Xubuntu 17.10 (artful)
  • gzip version: 1.6
  • pigz version: 2.4
  • The file being compressed is 9.25 GiB SQL dump

gzip quick

time gzip -1kN ./db_dump.sql

real    1m22,271s
user    1m17,738s
sys     0m3,330s

gzip best

time gzip -9kN ./db_dump.sql 

real    10m6,709s
user    10m2,710s
sys     0m3,828s

pigz quick

time pigz -1kMN ./db_dump.sql 

real    0m26,610s
user    1m55,389s
sys     0m6,175s

pigz best (no zopfli)

time pigz -9kMN ./db_dump.sql 

real    1m54,383s
user    14m30,435s
sys     0m5,562s

pigz + zopfli algorithm

time pigz -11kMN ./db_dump.sql 

real    171m33,501s
user    1321m36,144s
sys     0m29,780s

As a bottomline I would not recommend the zopfli algorithm since the compression took tremendous amount of time for a not-that-significant amount of disk space spared.

Resulting file sizes:

  • bests: 1309M
  • quicks: 1680M
  • zopfli: 1180M
Improve this answer
0
3

The LZMA2 compressor of p7zip Install p7zip uses both cores on my system.

Improve this answer
0

Relevant Arch Wiki entry: https://wiki.archlinux.org/index.php/Makepkg#Utilizing_multiple_cores_on_compression

# lzma compression
xz --threads=0

# drop-in parallel gzip replacement
# -p/--processes flag can be used to employ less cores
pigz

# drop-in parallel bzip2 replacement
# -p# flag can be used to employ less cores
# (note: no space between the -p and number of cores)
pbzip2

# modern zstd compression
# is used to build Arch packages by default
# since somewhere 2020
zstd --threads=0
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.