It has been determined already in this question that tar cannot read input from stdin.

How else can a dd output be archived directly possibly without using any compression? The purpose of doing everything in a single task is to avoid to write the dd output to the target disk twice (once as a raw file and once as an archive), and to avoid to perform two different tasks, which is a waste of time (since the input file must be read and written and the output read, processed and written again), and can be impossible if the target drive is almost full.

I'm planning to do multiple backups of drives, partitions and folders, and I'd like to benefit both from the ease of having everything stored into a single file and from the speed of each backup / potential restore task.

  • 1
    I assume you're aiming for: dd if=/dev/sdXY | tar ..., but why not tar directly: tar cf foo.tar /dev/sdXY? – muru Feb 18 '15 at 20:29
  • 1
    @murru From your last answer i understood that since the file at that point of the pipeline is just a raw binary in stdout and has not a proper entry into the filesystem it cannot be archived directly by simply piping it to tar. Anyway my bad, i wrote partitions but i actually want this to work for whole drives, and that's the reason why i need to use dd first – kos Feb 18 '15 at 21:07
  • 1
    @muru The difference is tar just packs all the files together, while i need to keep the partition scheme as well – kos Feb 18 '15 at 21:27
  • 1
    @muru serverfault.com/questions/478674/linux-backup-using-tar Look at the top answer. tar does not backup MBR/GPT, since those sectors are outside of the filesystem "scope", therefore no partition table is backed up, and you'll need to recreate the partition scheme manually once you made a back up with tar, while with dd there's no such need – kos Feb 19 '15 at 8:32
  • 1
    @muru Ok, so you are implicitly stating that tar does back up MBR/GPT right? So how would you restore from a backup? Let's say i tar cf sda.tar /dev/sda and move the tarball to sdb, then i switch the original sda disk for a newer one of the same capacity. This disk comes of course with just unallocated space. How would you restore starting from this point? – kos Feb 19 '15 at 8:46

If you want to dump a whole block device to a file, tar won't be of any use, because it doesn't work with block devices. Instead you'll need to use dd or similar:

dd if=/dev/sdX of=/path/to/backup bs=16m

Even like this it would be better to use at least a little compression, as long as it doesn't slow down the transfer too much. In short, you need a compression algorithm with a throughput not much lower than that of your slowest storage medium. There are several such compression algorithms. The most notorious are Lempel–Ziv–Oberhumer, its derivate L4Z, and Snappy. There's a comparison of various compression algorithms including those three on the L4Z project page:

Name            Ratio  C.speed D.speed
                        MB/s    MB/s
LZ4 (r101)      2.084    422    1820
LZO 2.06        2.106    414     600
QuickLZ 1.5.1b6 2.237    373     420
Snappy 1.1.0    2.091    323    1070
LZF             2.077    270     570
zlib 1.2.8 -1   2.730     65     280
LZ4 HC (r101)   2.720     25    2080
zlib 1.2.8 -6   3.099     21     300

For the sake of this answer, I'll choose an example with LZO, because it's readily available in Canonical's repositories in the form of lzop, but ultimately all those stream compressors have front-ends that read from standard input and write to standard output.

dd if=/dev/sdX bs=16m | lzop > /path/to/backup.lzo

If you want to work on the same machine during backup, you may want to use ionice and/or nice/schedtool:

ionice -c 3 dd if=/dev/sdX bs=16m | ionice -c 3 schedtool -B -n 10 -e lzop > /path/to/backup.lzo
  • Thanks, i'll take a look even at the other alternatives. – kos Mar 8 '15 at 22:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.