I'm looking for a deduplicating copy-on-write filesystem solution for general user data such as /home and backups of it. It should use online/inline/synchronous deduplication at the block-level using secure hashing (for negligible chance of collisions) such as SHA256 or TTH. Duplicate blocks need not even touch the disk.

The idea is that I should be able to just copy /home/<user> to an external HDD with the same such filesystem to do a backup. Simple. No messing around with incremental backups where corruption to any of the snapshots will nearly always break all later snapshots, and no need to use a specific tool to delete or 'checkout' a snapshot. Everything should simply be done from the file browser without worry. Can you imagine how easy this would be? I'd never have to think twice about backing-up again!

I don't mind a performance hit, reliability is the main concern. Although, with specific implementations of cp, mv and scp, and a file browser plugin, these operations would be very fast, especially when there is a lot of duplication as they would only need to transfer the absent blocks. Accidentally using conventional copy tools that do not integrate with the FS would merely take longer, waste some bandwidth when copying remotely and waste some CPU, as the duplicate data would be re-read, re-transferred and re-hashed (although nothing would be re-written), but would absolutely not corrupt anything. (Some filesharing software may also be able to benefit by integrating with the FS.)

So what's the best way of doing this?

I've looked at some options:

  • lessfs - Looks unmaintained. Any good?
  • Opendedup/SDFS - Java? Could I use this on Android?! What does SDFS stand for?
  • Btrfs - Some patches floating around on mailing list archives, but no real support.
  • ZFS - Hopefully they'll one day relicense under a true Free/Opensource GPL-compatible licence.

Also, 2 years ago I had a go at an attempt in Python using Fuse at the file-level to be used over the top of a typical solid FS such as EXT4, but I found Fuse for Python underdocumented and didn't manage to implement all of the system calls.

  • Not a file system, but you might want to look at the recently released backup tool Obnam. – JanC Jun 10 '12 at 23:35
  • @JanC: Thanks, it definitely looks like backup software done right! It avoids this: "No messing around with incremental backups where corruption to any of the snapshots will nearly always break all later snapshots," ...but not this: "and no need to use a specific tool to delete or 'checkout' a snapshot.". Definitely a good find though! I might be able to work with some of the Python code. :-) – James Haigh Jun 11 '12 at 0:18
  • @JamesHaigh You may want to post this idea in Ubuntu BrainStorm too; if you have not done it already. – Samik Jun 11 '12 at 10:23
  • @JamesHaigh one of the things on the Obnam roadmap is a fuse-based filesystem (but I don't think there is a timeline showing when to expect that). That would bring "no additional tool" closer. – JanC Jun 16 '12 at 18:39
  • If you want to avoid the corruption of one backup breaking latter ones, you simply do a multi generational incremental backup. Instead of doing a full on sunday and a daily differential that is always relative to the day before, you for instance, on Wednesday do an incremental backup relative to the full one on sunday ( rather than Tuesday ). Then if Tuesday is corrupt, you don't care since if you have Wednesday's backup, you skip Monday and Tuesday when restoring. – psusi Jun 4 '13 at 16:02

This sounds very enterprise (as in pricey).

datadomain offers data de-duplication, and maybe netapp with their wafl filesystem. But at a high cost.

A "free" alternative could be zfs.

According to me though the "best" and most Linuxy alternative, although on a file level instead of "block level", would be rsnapshot. It uses rsync and hardlinks to manage versioning.

I rather trust old proven tools than using a new filesystems like Btrfs which hasn't been around long enough for people to discover all kinds of nasty bugs.

  • Enterprise? No, it's more the kind of thing I'd expect to see by default in Ubuntu in the future due to it's plain ease of use. I did mention ZFS but it's CDDL so not Free with a capital F. rsnapshot is not a filesystem, it's more of a hack to bring snapshot-like functionality to legacy filesystems, doesn't create 'real' snapshots (snapshots are supposed to be atomic), and is incremental (see 2nd paragraph). I'd rather use a relatively new FS than something that uses incrementals. – James Haigh Jun 10 '12 at 23:29
  • I guess 'file-level' is a little ambiguous, but I've seen the term used in the context of filesystems along with block-level, byte-level and bit-level. This was the context I meant it as. – James Haigh Jun 10 '12 at 23:47
  • 1
    @JamesHaigh According to the Free Software Foundation, the CDDL is a free (as in freedom) license, it's just not GPL-compatible. This means CDDL's code (like the official ZFS implementation) and GPL's code (like the Linux kernel) cannot legally be linked together, and since drivers in Linux are generally linked as modules, that hinders CDDL support in Linux-based operating systems. – Eliah Kagan Jun 11 '12 at 2:22
  • 2
    ZFS on Linux possibly (depends who you ask) gets around this by being separately distributed. Alternatively, FUSE lets filesystem "drivers" operate in userspace where they need not themselves link to the kernel, and there is a FUSE version of ZFS as well. So you have multiple options for using ZFS filesystems on operating systems like Ubuntu that use Linux as their kernel. (Some OSes, like Debian, let you use a different kernel.) – Eliah Kagan Jun 11 '12 at 2:24

I'm looking and investigating exactly the same, I could sugget https://attic-backup.org/quickstart.html#automating-backups for now, seems to be quiet simple and good for backups of linux.

There is also bacula with this feature, but attic seems to be good enough for most cases.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.