Gzip time to decompress?

Question

I wanted to backup my entire server, which is hosted. So I used dd and gzip to make the file smaller. The disk was 500GB, but with less than 5% used. I managed to dd the whole partition into a 200 GB gzip file over the internet to my home in 8 hours. Now I am trying to decompress the file into a partition onto a new disk. It has already taken more than 8 hours and of course I have no way of determining progress.

1. How long should it take to decompress in relation to compressing?
2. I assume the local processor is the key ingredient in determining how long it takes? (Rather than the bandwidth of the network)
3. Is there a way of seeing progress?

Can I do this a better way next time?

Answer 1

You don't want to (ab)use dd like that. It will waste time copying the 95% of the disk that is not being used, and you will get a corrupt image if you have it mounted read/write at the time. If you want to backup the system, it is best to make sure you shut down all services that could be writing to the disk, and use tar.

Answer 2

Here's an article I saw with some benchmarks for gzip and some other compression algorithms: http://tukaani.org/lzma/benchmarks.html. I would assume that decompression time varies with your CPU speed. Also, if you look through the tests, it seems like decompression is almost always faster than compression.

Edit:

In response to your latest question about other ways to back up your server, I found this article that talks about various backup methods: http://www.techrepublic.com/blog/10things/10-outstanding-linux-backup-utilities/895. I'm not sure what access you have to the server, but if it is a common commercial host, you might be able to ask Tech Support how you should do it.

Answer 3

The simplest way to look at the progress of a running gzip process is to simply look at the file size of the written file, potentially in combination with watch if you want real time updates. If you are dealing with partitions that is of course not easily possible.

An alternative to estimate the progress is to use iotop. iotop will show you the speed at which data is written to the disk by every process on the system, your gzip process will likely show up on top and give you the amount of data processed per second. Then simply multiply the MB/sec with how long the process has been running (see ps auxw START column) and you will get a rough idea how long it will take.

As for further backup runs: Use rsync when you want to copy data from one computer on a network to another. rsync handles compression and deltas, thus you only have to transfer the data that you don't already have, which makes it very fast for regular updates. rsync also has the --backup and --backup-dir options that can be used to create not just copies, but proper backups that keep track of deleted files.

And when doing diskimages partimage is a good alternative to dd, as unlike dd partimage is file system aware and will only copy blocks actually used by the file system, not empty unused blocks, it is thus able to create much smaller disk images on mostly empty file systems. But it's not a good tool for backup over a network either, use rsync instead.

Answer 4

Your hoster should have provided you with a backup plan and (most preferably) a web-interface to the backup and restore facilities. The better way would be to ask them if they let you access a copy of an automated backup.

Answer 5

Generally decompressing in gzip should be faster than compressing. I suspect the problem here is that the target disk is slower than the disk from which it was compressed: or perhaps you're reading from and writing to the same physical disk which is causing a lot of seeking.

Other answers are correct that generally it's better to back up the files, rather than the raw device.

To see progress, I would install pv and then say something like this:

zcat /tmp/myimg.gz |pv -s500G > /tmp/myimg