Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I wanted to backup my entire server, which is hosted. So I used dd and gzip to make the file smaller. The disk was 500GB, but with less than 5% used. I managed to dd the whole partition into a 200 GB gzip file over the internet to my home in 8 hours. Now I am trying to decompress the file into a partition onto a new disk. It has already taken more than 8 hours and of course I have no way of determining progress.

  1. How long should it take to decompress in relation to compressing?
  2. I assume the local processor is the key ingredient in determining how long it takes? (Rather than the bandwidth of the network)
  3. Is there a way of seeing progress?

Can I do this a better way next time?

share|improve this question
    
Seems to be a rather inefficient way of doing a copy, provided you have only 25Gb of data - i.e. likely less than 20Gb gzipped :) –  Sergey Sep 26 '11 at 14:13
    
If you already downloaded it, why would the network bandwidth be relevant? –  poolie Sep 26 '11 at 23:42

5 Answers 5

You don't want to (ab)use dd like that. It will waste time copying the 95% of the disk that is not being used, and you will get a corrupt image if you have it mounted read/write at the time. If you want to backup the system, it is best to make sure you shut down all services that could be writing to the disk, and use tar.

share|improve this answer

Here's an article I saw with some benchmarks for gzip and some other compression algorithms: http://tukaani.org/lzma/benchmarks.html. I would assume that decompression time varies with your CPU speed. Also, if you look through the tests, it seems like decompression is almost always faster than compression.

Edit:

In response to your latest question about other ways to back up your server, I found this article that talks about various backup methods: http://www.techrepublic.com/blog/10things/10-outstanding-linux-backup-utilities/895. I'm not sure what access you have to the server, but if it is a common commercial host, you might be able to ask Tech Support how you should do it.

share|improve this answer
    
Thanks, that is very useful. Do you know of any way I can determine if it is still progressing? The target disk is unmounted and I assume mounting it would halt the progress? How do I see if the process is still active? I can see the drive light flashing, but that does not really indicate progress. –  Laurence Sep 26 '11 at 14:42
    
I think I've used dd on a flash drive that was mounted before.I could be wrong... I'm a little new to Linux. If you can, try mounting the drive and seeing GB's have been copied? That should give you information on how much is left. –  ThatOtherPerson Sep 26 '11 at 14:46
    
Thanks, according to the benchmark, my 200 GB disk will take 38 days to decompress!!! 200GB/61MB/S –  Laurence Sep 26 '11 at 15:03
    
200GB at 61MB/sec it will only take around hour, not 38 days. –  Grumbel Sep 26 '11 at 15:44

The simplest way to look at the progress of a running gzip process is to simply look at the file size of the written file, potentially in combination with watch if you want real time updates. If you are dealing with partitions that is of course not easily possible.

An alternative to estimate the progress is to use iotop. iotop will show you the speed at which data is written to the disk by every process on the system, your gzip process will likely show up on top and give you the amount of data processed per second. Then simply multiply the MB/sec with how long the process has been running (see ps auxw START column) and you will get a rough idea how long it will take.

As for further backup runs: Use rsync when you want to copy data from one computer on a network to another. rsync handles compression and deltas, thus you only have to transfer the data that you don't already have, which makes it very fast for regular updates. rsync also has the --backup and --backup-dir options that can be used to create not just copies, but proper backups that keep track of deleted files.

And when doing diskimages partimage is a good alternative to dd, as unlike dd partimage is file system aware and will only copy blocks actually used by the file system, not empty unused blocks, it is thus able to create much smaller disk images on mostly empty file systems. But it's not a good tool for backup over a network either, use rsync instead.

share|improve this answer

Your hoster should have provided you with a backup plan and (most preferably) a web-interface to the backup and restore facilities. The better way would be to ask them if they let you access a copy of an automated backup.

share|improve this answer

Generally decompressing in gzip should be faster than compressing. I suspect the problem here is that the target disk is slower than the disk from which it was compressed: or perhaps you're reading from and writing to the same physical disk which is causing a lot of seeking.

Other answers are correct that generally it's better to back up the files, rather than the raw device.

To see progress, I would install pv and then say something like this:

zcat /tmp/myimg.gz |pv -s500G > /tmp/myimg
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.