Ubuntu 18.04 LTS - LXD Ubuntu launch using loopback ZFS (about 400MB)
# lxc launch ubuntu:16.04 test -s ianzfspool
(I have one other idle multi-GB container using the same ianzfspool.) I have lots of space in the pool:
# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
ianzfspool 99.5G 13.7G 85.8G - 2% 13% 1.00x ONLINE -
If I loop, using lxc
to create snapshots of the idle test container (400MB), the first ones create in a few seconds.
As the number of lxc
snapshots grows (currently over 1,200), now they take minutes to create. I have a script running in the test container that puts the current date in a file in /tmp every few seconds, so I have one file that is changing in the container in every snapshot; otherwise, the container is mostly idle. The first lxc
snapshots used about 25K; the current ones (after 1,200 snapshots) use 66K. (After 2,470 snapshots they each use 115K and after 7,500 snapshots they each use 293K.)
#!/bin/sh -u
# Create snapshots of the Ubuntu test container.
count=0
while : ; do
count=$(( count + 1 ))
cp=$( printf "%05d" $count )
lxc snapshot test snap$cp
echo "$0: done $cp" >/tmp/icount.txt
done
Edit 1: I currently am up to about 2,470 lxc
snapshots and each new snapshot is taking about four minutes to create and uses about 115K. If I call zfs snapshot
to take a snapshot of the container directly, instead of using lxc snapshot
, the snapshot takes less than a second, even with 2,470 existing snapshots. The direct zfs
snapshot only uses 20K (instead of 115K). Running zfs list
takes only a second or two. If I run lxc list
(instead of zfs list
), that currently takes over four minutes with 2,470 existing snapshots. So the snapshot creation and listing slowdown isn't with ZFS, it's with LXD. Indeed, the lxd
process itself has used 4,253 CPU seconds and has a VSIZE of 3.9G so far in this snapshot creation experiment.
I paused the lxc
snapshot creation script and wrote one to loop creating direct zfs
snapshots, and it has created more snapshots in five minutes than lxc
has in three days. I paused it after about 3,000 direct ZFS snapshots.
I just re-ran lxc list
and of course it still only shows the 2,471 lxc
snapshots, but now it ran in only 40 seconds instead of four minutes. I just created another lxc snapshot
and it only took 49 seconds instead of four minutes. What changed? Has creating a bunch of direct zfs
snapshots somehow sped up lxc
snapshot creation and listing? The lxc
snapshots are still way slower to create than direct zfs
snapshots, but something has improved with lxc
snapshot creation (from four minutes to 49 seconds).
I let the ZFS snapshot creation proceed until 10,000 direct ZFS snapshots were created. (So now I have 2,472 lxc
snapshots and 10,000 direct zfs
snapshots of this test container.) The lxc list
now takes 90 seconds (instead of 40) and lxc snapshot
takes 100 seconds (instead of 49). Using ZFS directly is still two orders of magnitude faster than going via LXD.
Edit 2: After a reboot, LXD snapshot creation has sped up. I'm at 7,470 snapshots created via my lxc snapshot
loop and each snapshot is now taking about 30 seconds to create and has a size of 293K. The lxc list
takes 45 seconds. The zfs list
takes 105 seconds and produces 17,507 lines of output (which includes the 10,000 direct ZFS snapshots). Doing zfs snapshot ianzfspool/containers/test@iansnap10001
directly (not via LXD) takes less than half a second -- still much faster than via LXD.
Where can I find documentation on why LXD snapshot creation is slowing down, and how I might speed things up? (If LXD with ZFS snapshots were as cheap as I was led to believe, I was hoping to run snapshots every 5 minutes and keep them for about a week, which would be 2,016 snapshots. I've read that LVM snapshots are a performance penalty, but nothing I've read said the same about LXD with ZFS. I see that I could bypass LXD and use ZFS snapshots, but why do I need to do that? Is LXD fixable?)
The lxd
process on the host is using a lot of cpu and is getting big (this after 1,200 snapshots):
PID VSTACK VSIZE RSIZE PSIZE VGROW RGROW SWAPSZ MEM CMD 1/8
10663 132K 3.7G 1.1G 0K 0K 0K 2580K 14% lxd