20

If I make a disk image and compress it with gzip/xz/etc is there a way to mount it directly without first uncompressing it?

Say I've used

sudo dd if=/dev/sdc1 | gzip -9 > image1.dd.gz

how can I mount the original image, without creating an uncompressed copy first?

Or I've used

sudo dd if=/dev/sdc | gzip -9 > wholedisk.dd.gz

and the disk has multiple partitions, would that make it any harder?

With an uncompressed image of a whole disk then using kpartx or newer versions of losetup with it's -P flag should create a loop for each partition.

But is there a way to mount/losetup/read the compressed image?

If it won't work for gzip/xz, is there any compression method this would work for?


Note on duplicate Q

The currently suggested duplicate

DOES NOT USE COMPRESSION, and IS NOT A DUPLICATE.

mount will not mount a compressed image by itself.

14
  • I can tell you that it definitely is possible. But I can't tell you how atm because the instructions are on 2 external HDD and right now I have access to neither but I want to let you know that it works. I'll come back with an answer when I have one (and no one else posted one until then).
    – UTF-8
    Oct 12, 2016 at 20:03
  • 3
    Possible duplicate of Mount single partition from image of entire disk (device)
    – Panther
    Oct 12, 2016 at 20:20
  • 1
    @UTF-8 That's encouraging, thanks. FYI, original goal is to make a smaller backup of a drive/partition that's nearly empty or has lots of easily compressible data, and be able to read it without writing a LOT of extra data or needing a ton of free space to extract it
    – Xen2050
    Oct 12, 2016 at 20:20
  • See also major.io/2010/12/14/…
    – Panther
    Oct 12, 2016 at 20:21
  • 2
    @bodhi.zazen Pardon? Maybe I should clarify, mount does not automatically uncompress images, so this Q is not a duplicate of askubuntu.com/questions/69363 (maybe I missed the joke, hopefully uncompressing isn't the only answer :)
    – Xen2050
    Oct 12, 2016 at 20:40

2 Answers 2

12

You can use squashfs to compress disk images and then mount them.

Create the disk image

If you haven't got a disk image yet use dd to create one:

dd if=/dev/sda1 of=/tmp/sda1.img bs=4M

Compress the image with squashfs

Install squashfs:

apt-get install squashfs-tools

Compress the image:

mksquashfs /tmp/sda1.img /tmp/sda1.squash.img

Or Stream the compression (don't need a temporary dd file)

mkdir empty-dir
mksquashfs empty-dir squash.img -p 'sda_backup.img f 444 root root dd if=/dev/sda bs=4M'

Compliments to terminator14 at UbuntuForums.org. Definitions/Explanations:

  • empty-dir - "source" dir. Basically in our case, just an empty dir to satisfy mksquashfs' input arg format
  • squash.img - the destination and filename of the output squashfs file
  • sda_backup.img - the name of the dd backup INSIDE the squashfs file
  • f - specifies that sda_backup.img is a regular file (as opposed to a directory, block device, or char device)
  • 444 - permissions of the sda_backup.img file inside the squashfs image
  • root root - UID and GID for the sda_backup.img file inside the squashfs image. Can be specified by decimal numbers, or by name
  • dd if=/dev/sda bs=4M - the dd command used to read the device we want backed up

Mount the image

First mount the squashfs image:

mkdir /mnt/squash
mount /tmp/sda1.squash.img /mnt/squash

This will present the un-compressed disk image for you to mount:

mkdir /mnt/sda1
mount /mnt/squash/sda1.img /mnt/sda1

Or if it's a full drive image (partitioned) you could use losetup to attach the dd image to a loop device (possibly optional) and then kpartx -a or partprobe to find & separate the partitions to separate devices, or even vgscan / vgchange -ay if there's LVM.

4
  • 2
    I added some on-the-fly squashfs info, to avoid needing a temporary plain dd file. Great answer with squashfs, just doesn't do the more pipe-friendly xz/gz formats
    – Xen2050
    Dec 11, 2016 at 11:25
  • Would that somehow work with a full disk image containing multiple partitions as well? Nov 18, 2017 at 17:43
  • See here for full disk: unix.stackexchange.com/questions/31669/…
    – HackerBoss
    Mar 19, 2020 at 15:53
  • Depending on the filesystem type, mounting may require the ro,noload options, like so: mount -o ro,noload /mnt/squash/sda1.img /mnt/sda1
    – HackerBoss
    Mar 19, 2020 at 16:58
9

I was able to get an NBD server to "serve up" an un-compressed version of a .gz or .xz file. I'm not sure if it has to read the entire compressed file very often (only tested so far with small files in ram) but at least it works without having to extract a whole un-compressed copy.

Viewing a gz / xz Compressed dd image "on-the-fly"

1 - Install nbdkit & client

apt install nbdkit nbd-client

2 (Optional for testing only) - Create demo "dd" style file - an empty 50M file with an ext filesystem & zip it

fallocate -l 50M 50m
mkfs.ext3 -v 50m

Then compress with either gzip or xz

gzip -vk9 50m
xz -v9k 50m

Note: An xz option of --block-size=16MiB should give better random access performance, but at the expense of less compression (maybe 1%?). See the nbdkit's xz plugin description.

3 - Run server on COMPRESSED image

nbdkit -v --no-fork -i 127.0.0.1 /usr/lib/i386-linux-gnu/nbdkit/plugins/nbdkit-gzip-plugin.so file=~/Downloads/50m.gz

or for xz

nbdkit -v --no-fork -i 127.0.0.1 /usr/lib/i386-linux-gnu/nbdkit/plugins/nbdkit-xz-plugin.so file=~/Downloads/50m.xz

Apparently newer versions of ndbkit can specify --filter=xz instead of specifying a whole plugin path:

nbdkit -v --no-fork -i 127.0.0.1  --filter=xz file=~/Downloads/50m.xz
  • The --no-fork tells the server "Don't fork into the background" so seeing possible error messages & killing it with CTRL-C is a little easier.

  • -v may / will show too many status messages, especially when reading.

4 - Run client (in another terminal/window)

sudo nbd-client -nofork 127.0.0.1 10809 /dev/nbd0

The -nofork is similar to -no-fork above, but I could not kill the client with CTRL-C, kill <pid>, or sudo kill <pid>. Ending the server with CTRL-C stopped both server & client.

5 - Observe that /dev/nbd0 is now identical to UN-COMPRESSED image

$ sudo diff -s /dev/nbd0 ~/Downloads/50m
Files /dev/nbd0 and /home/user/Downloads/50m are identical

/dev/nbd0 can now be used just like the uncompressed image; mounted, read/copy with dd, or use kpartx, partprobe, vgscan / vgchange, etc!

Thanks to Cristian Ciupitu on Unix & Linux, though it took a little more digging to find the plugin & get everything going together.

4
  • On Fedora 31 nbdkit -v --no-fork -i 127.0.0.1 --filter=xz file ~/Downloads/50m.xz worked great with the same client command
    – EkriirkE
    Sep 7, 2019 at 10:51
  • On Ubuntu xenial 16.04 LTS nbdkit 1.1.11 uses old NBD protocol, which differs from new NBD protocol, used in nbd-client 3.13, but you ca use nbd-client version before 3.10, e.g. nbd-client 3.8 from Ubuntu precise 12.04 old LTS [use old versions only locally for security!] Dec 25, 2019 at 12:32
  • Simply specifying xz in place of /usr/lib/.../nbdkit-xz-plugin.so works great and lets it pick x86_64-linux-gnu or i386-linux-gnu automatically. Feb 6, 2021 at 5:27
  • 1
    As of Ubuntu 20.04 (nbdkit v1.16.2), the nbdkit-xz-plugin.so file is missing. So, probably one should use --filter=xz as specified by @EkriirkE. Whereas, I simply downloaded the deb for nbdkit for Debian Buster, extracted the plugin from it and used it, and that worked too. Feb 6, 2021 at 5:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .