3

For some reason there don't seem to be too many questions or answers about using dd for creating "smaller" image files with the conv=sparse option here.

So when using dd for image creation, can I safely use e.g. dd if=/dev/sda of=/media/storage-device/sda.img conv=sparse to get an image file that uses (much) less space on the target?

Don't know if it matters, but in this case the source would be a SSD with EXT4 filesystem and only 15% used (as it is quite new), and the target also would use EXT4 fs.

Or is this possibly a rather bad idea (under certain circumstances) and may lead to problems (later, e.g. when I need to restore it)?

I am also wondering if this would work when the target is a network mount or filesystem?

The goal would be to get an image file of either a partition or (even better) a whole disk that doesn't take much more space than needed, while keeping it easily mount- and usable.

Extra compression isn't really needed or wanted, and it usually also would make the resulting image not mountable any more I guess, so I can live without it.

And I know Clonezilla can do stuff like saving only used sectors, but I don't like its limitations, e.g. Due to the image format limitation, the image can not be explored or mounted. (And of course dd is already available on every system.)

Many thanks in advance

6
  • If the disk usage is really that low, even simple gzip compression will likely give a small image and that is definitely safe.
    – muru
    Commented Oct 14, 2021 at 2:51
  • 1
    Compression or using things like conv=sparse will only be efficient, if you zeroise the free space before. It can be done with zerofree with ext file systems or generally with dd by creating a huge file filled with zero-bytes and then remove that file. This is a slow process, so I prefer Clonezilla, but I understand that you want to explore this process. Anyway, I don't know if the method you suggest will work, and I'm looking forward to your result :-P
    – sudodus
    Commented Oct 14, 2021 at 8:27
  • 1
    Otherwise, an obvious alternative is to work on the file level, use tar with or without compression to get a useful 'image' of a partition. This is not a true image, but can be very useful in order to store the content, so that it can be restored to the same and even to a smaller partition.
    – sudodus
    Commented Oct 14, 2021 at 8:38
  • 1
    I see that every approach has some advantage (but also downsides) Probably I could pipe the output of dd to gzip or so, and so the target filesystem doesn't even have to support sparse files and the resulting image file may even be smaller - but in order to be able to mount it or restore single files from it, I would first have to extract it somewhere with plenty of free disk space. And if the archive's header gets corrupted for some reason, it will make the whole image entirely useless because I couldn't uncompress it any more I guess. So one should probably store a 2nd image somewhere else
    – user79
    Commented Oct 14, 2021 at 14:21
  • 1
    "will only be efficient if you zeroise the free space before" - In this case this shouldn't be neccessary as the drive is still pretty new, but generally speaking you're right, so thanks for the info. I just thought this kind of would be done automatically on SSDs by trim command...? - I probably will end up creating one "sparse" image, a somehow compressed one, and maybe a tar archive, just in case one file gets corrupted or can't be restored correctly for some reason :D
    – user79
    Commented Oct 14, 2021 at 14:51

2 Answers 2

Reset to default
5

The easiest method I've found to accomplish this is to trim all partitions on the drive with the appropriate tool (fstrim on Linux, whatever the tool is called on windows) then dump the gpt partition table to a file using gdisk and dd each partition through a compressor (I usually use zstd for speed) and save the output compressed images alongside the partition table dump. You'll compress each partition image down a bit under the size of used space and it's trivial to blast the images back onto an empty drive later. I do this with every brand new laptop to take a clean factory image before I nuke the drive and install Linux, if I need to send the machine for warranty repair or decide to sell it it's simple to restore it to factory state.

1

Cannot answer if conv=sparse is wise/safe to use, but you will get a sparsified result using dd together with compression (I use zstd) and if you "prepare" the source-disk.

I took an image of a 256GB SSD with Win 11 installed.

In Windows I trimmed the NTFS partitions, wrote zero to all free space and then did a trim once again:

Trimming: defrag C: /L

Zero free space inside Windows: sdelete -z C: (from sysinternals)

I made a image from Linux with dd and ztd:

dd bs=1024K if="/dev/nvme0n1" | pv --size 256060514304 | zstd --quiet --force -o "/mnt/data/image.img.zst"

The image was written and compressed with about about 150-350MB/s.

After that, when running into the empty space of the source-disk, data was processed at 1.2GBs :-) And the image-file was only growing with about 100KB for the remaining 220GB free space.

I would says that's just as good as "real" sparse. The size of the imagefile was 19.3GB. The W11 installation size was 48GB (if you delete the page-file on the Windows installation you might save some additional MBs/GBs).

To restore image:

zstd --quiet --stdout --decompress "/mnt/disk1/image.img.zst" | pv | dd bs=1024K of="/dev/nvme0n1"

Extra info:
To zero out a mounted partition in Linux cat /dev/zero | pv > /mnt/dis1/zero.dat.
And another example: W10 LTSC, size of installation (C:) 18.2GB (no pagefile):
Imagesize with dd: 6.26GB
Imagesize with dism: 6.28GB

1
  • If you just delete the pagefile, space is not saved with sparse. It is necessary to turn the free space to zeros in Windows (a lot of zero bytes). It means: delete the pagefile, then create a file full of zeros up to entire free space, and then delete it.
    – Milan Kerslager
    Commented Jul 6 at 22:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .