I'm thinking about using btrfs functions for both deduplicate and compress my storage...

My question is: is it efficient?

I mean, does it deduplicate AND compress at the same time efficiently?!

I'm using Ubuntu 14.04 64-bit. Also, I'm planning to use it with raid0 on top of EBS-Optimized volumes in a Cloud Computing environment...

Tks! Thiago

  • I'd not put anything that I am not willing to lose on btrfs at this time....not mature enough. – mdpc May 31 '14 at 22:48

If you are writing about the deduplication with the bedup utility, then the answer is: deduplication and compression are completely independent processes.

During the deduplication, no file contents are rewritten (only metadata), so there is no opportunity to compress the data.

And I am 99% sure, that compression status is transparent to the bedup since it is next to impossible to get the original, compressed contents of the file out of the btrfs.

On the other hand, recompression of old data happens when you defragment the drive (e.g. with btrfs filesystem defragment -r -v -clzo /). This will currently break the de-duplication of files.

  • Defragmentation undeduplicates (re-duplicates?) the data though, doesn't it? Because the cow-ed files are no longer cow-ed, if I understand correctly. Or are you saying this changed with 3.9? – Jo-Erlend Schinstad Jun 1 '14 at 9:54
  • After kernel 3.9 defregmentation no longer re-duplicate files. Or so it is says - I never tested this aspect myself. The particular kernel patch (git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/…) says only about subvolumes. It seems that defragmentation inside a subvolume never resulted in reduplication of deduplicated contents. – Adam Ryczkowski Jun 2 '14 at 10:55
  • 1
    Ah! That's really nice! :) I find it slightly difficult to keep up with Btrfs development. :) – Jo-Erlend Schinstad Jun 6 '14 at 14:05

There is no online deduplication currently. It's planned, but not yet implemented. There is an experimental patchset ( www.mail-archive.com/linux-btrfs%40vger.kernel.org/msg32862.html ) if you want to try it.

Just from personal experience there is no noticeable performance hit of online compression on a hasswell i5 4460 when writing to a ssd and 3 hdds at once (one of the hdds is formated with xfs so no compression was being performed. Defragging/recompressing the 3 btrfs drives gives a cpu load around 50% but the system still feels smooth. This is with lzo and compression not forced (unless you force it btrfs stops trying to compress a file if it's incompressible + lzo is faster than zlib). The limit is still drive speed, even with a ssd.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.