I was able to get an NBD server to "serve up" an un-compressed version of a .gz or .xz file. I'm not sure if it has to read the entire compressed file very often (only tested so far with small files in ram) but at least it works without having to extract a whole un-compressed copy.
Viewing a gz / xz Compressed dd image "on-the-fly"
1 - Install nbdkit & client
apt install nbdkit nbd-client
2 (Optional for testing only) - Create demo "dd" style file - an empty 50M file with an ext filesystem & zip it
fallocate -l 50M 50m
mkfs.ext3 -v 50m
Then compress with either gzip or xz
gzip -vk9 50m
xz -v9k 50m
Note: An xz
option of --block-size=16MiB
should give better random access performance, but at the expense of less compression (maybe 1%?). See the nbdkit's xz plugin description.
3 - Run server on COMPRESSED image
nbdkit -v --no-fork -i 127.0.0.1 /usr/lib/i386-linux-gnu/nbdkit/plugins/nbdkit-gzip-plugin.so file=~/Downloads/50m.gz
or for xz
nbdkit -v --no-fork -i 127.0.0.1 /usr/lib/i386-linux-gnu/nbdkit/plugins/nbdkit-xz-plugin.so file=~/Downloads/50m.xz
Apparently newer versions of ndbkit can specify --filter=xz
instead of specifying a whole plugin path:
nbdkit -v --no-fork -i 127.0.0.1 --filter=xz file=~/Downloads/50m.xz
The --no-fork
tells the server "Don't fork into the background" so seeing possible error messages & killing it with CTRL-C is a little easier.
-v
may / will show too many status messages, especially when reading.
4 - Run client (in another terminal/window)
sudo nbd-client -nofork 127.0.0.1 10809 /dev/nbd0
The -nofork
is similar to -no-fork
above, but I could not kill the client with CTRL-C, kill <pid>
, or sudo kill <pid>
. Ending the server with CTRL-C stopped both server & client.
5 - Observe that /dev/nbd0
is now identical to UN-COMPRESSED image
$ sudo diff -s /dev/nbd0 ~/Downloads/50m
Files /dev/nbd0 and /home/user/Downloads/50m are identical
/dev/nbd0
can now be used just like the uncompressed image; mounted, read/copy with dd
, or use kpartx
, partprobe
, vgscan
/ vgchange
, etc!
Thanks to Cristian Ciupitu on Unix & Linux, though it took a little more digging to find the plugin & get everything going together.