2

I want to compress about 100'000 files (that's what find . -type f | wc -l says) with a total disk usage of 100 GB. Most of the files are small but just a handful of them makes up about 70 GB of the 100 GB.

I don't want to use tar or tar.gz for this because if I want to access the archive, File Roller first has to read in the entire archive from the external HDD before I can even see the file list. Same thing if I try to list the files on the terminal.

I don't need the rights management of tar because I can remember the few files which need other rights than the others. What compression algorithm should I use?

And while I'm at it: I make full disk backups with this command:

dd if=/dev/sda bs=32M | gzip -9 > /location/dateAndMachineName.gz

It does a pretty good compression. But do you know a better compression algorithm?

  • Use lzma or xz. – muru May 23 '15 at 13:56
  • @muru in combination with tar or not? Using it in combination with tar will result in just the same issue, or will it? – UTF-8 May 23 '15 at 14:16
  • They're tools just like gzip, but different algorithms. Whatever reasoning you applied to arrive at using gzip would apply to them too. – muru May 23 '15 at 14:17
  • @mure Then why would I use either lzma or xz instead of gz? – UTF-8 May 23 '15 at 14:21
  • 1
    Care to try SquashFS? It's a compressed, read-only filesystem that you can mount to see the contents directly. You'll see it used on Ubuntu live CDs, for example. tldp.org/HOWTO/SquashFS-HOWTO/creatingandusing.html – muru May 23 '15 at 14:50
4

The only solution I am aware of is pixz (sudo apt-get install pixz), a variant of xz using a blocked encoder which allows for fast random acccess/indexing. Additionally, it is a parallel method using multiple cores for compression.

Citing the docs:

The existing XZ Utils ( http://tukaani.org/xz/ ) provide great compression in the .xz file format, but they have two significant problems:

  • They are single-threaded, while most users nowadays have multi-core computers.
  • The .xz files they produce are just one big block of compressed data, rather than a collection of smaller blocks. This makes random access to the original data impossible.

With pixz, both these problems are solved.

Usage is simple:

tar -Ipixz -cf foo.tpxz foo to compress a folder foo

pixz -l foo.tpxz to list files in it (fast!)

pixz -x <file_path> < foo.tpxz | tar x to extract a single file given <file_path> in the archive

As a bonus, you will get access rights stored as well since the files are tarred first!

1

I con only think of one solution for you: Make a new partition, with a btrfs filesystem and activate transparent compression. Keep in mind tha some people still considder btrfs an "experimental" filesystem. That being said, my secondary backup HDD is using btrfs (for little over 2 years) and so far it's given me 0 issues. But as usual YMMV.

This and this should get you started with btrfs, if you are not familiar with it already.

  • 1
    +1: Your solution look for me highly interesting (in fact I'm sure that the OP will be also interested) Hower, would be a very good answer if* you include a step-by-step procedure. Link-only answers are not welcome. – Helio May 23 '15 at 19:52
  • I know about link answers, but for a step by step procedure I would need to know more about the OP's setup, such as HDDs, current partitions, etc, which I bealive is a bit out of scope... – Stunts May 23 '15 at 22:42
  • Ok, sorry. I understood your viewpoint. :-/ – Helio May 24 '15 at 10:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.