Ubuntu 18.04 LTS - LXD Ubuntu launch using loopback ZFS (about 400MB)
# lxc launch ubuntu:16.04 test -s ianzfspool
(I have one other idle multi-GB container using the same ianzfspool.) I have lots of space in the pool:
# zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT ianzfspool 99.5G 13.7G 85.8G - 2% 13% 1.00x ONLINE -
If I loop, using
lxc to create snapshots of the idle test container (400MB), the first ones create in a few seconds.
As the number of
lxc snapshots grows (currently over 1,200), now they take minutes to create. I have a script running in the test container that puts the current date in a file in /tmp every few seconds, so I have one file that is changing in the container in every snapshot; otherwise, the container is mostly idle. The first
lxc snapshots used about 25K; the current ones (after 1,200 snapshots) use 66K. (After 2,470 snapshots they each use 115K and after 7,500 snapshots they each use 293K.)
#!/bin/sh -u # Create snapshots of the Ubuntu test container. count=0 while : ; do count=$(( count + 1 )) cp=$( printf "%05d" $count ) lxc snapshot test snap$cp echo "$0: done $cp" >/tmp/icount.txt done
Edit 1: I currently am up to about 2,470
lxc snapshots and each new snapshot is taking about four minutes to create and uses about 115K. If I call
zfs snapshot to take a snapshot of the container directly, instead of using
lxc snapshot, the snapshot takes less than a second, even with 2,470 existing snapshots. The direct
zfs snapshot only uses 20K (instead of 115K). Running
zfs list takes only a second or two. If I run
lxc list (instead of
zfs list), that currently takes over four minutes with 2,470 existing snapshots. So the snapshot creation and listing slowdown isn't with ZFS, it's with LXD. Indeed, the
lxd process itself has used 4,253 CPU seconds and has a VSIZE of 3.9G so far in this snapshot creation experiment.
I paused the
lxc snapshot creation script and wrote one to loop creating direct
zfs snapshots, and it has created more snapshots in five minutes than
lxc has in three days. I paused it after about 3,000 direct ZFS snapshots.
I just re-ran
lxc list and of course it still only shows the 2,471
lxc snapshots, but now it ran in only 40 seconds instead of four minutes. I just created another
lxc snapshot and it only took 49 seconds instead of four minutes. What changed? Has creating a bunch of direct
zfs snapshots somehow sped up
lxc snapshot creation and listing? The
lxc snapshots are still way slower to create than direct
zfs snapshots, but something has improved with
lxc snapshot creation (from four minutes to 49 seconds).
I let the ZFS snapshot creation proceed until 10,000 direct ZFS snapshots were created. (So now I have 2,472
lxc snapshots and 10,000 direct
zfs snapshots of this test container.) The
lxc list now takes 90 seconds (instead of 40) and
lxc snapshot takes 100 seconds (instead of 49). Using ZFS directly is still two orders of magnitude faster than going via LXD.
Edit 2: After a reboot, LXD snapshot creation has sped up. I'm at 7,470 snapshots created via my
lxc snapshot loop and each snapshot is now taking about 30 seconds to create and has a size of 293K. The
lxc list takes 45 seconds. The
zfs list takes 105 seconds and produces 17,507 lines of output (which includes the 10,000 direct ZFS snapshots). Doing
zfs snapshot ianzfspool/containers/test@iansnap10001 directly (not via LXD) takes less than half a second -- still much faster than via LXD.
Where can I find documentation on why LXD snapshot creation is slowing down, and how I might speed things up? (If LXD with ZFS snapshots were as cheap as I was led to believe, I was hoping to run snapshots every 5 minutes and keep them for about a week, which would be 2,016 snapshots. I've read that LVM snapshots are a performance penalty, but nothing I've read said the same about LXD with ZFS. I see that I could bypass LXD and use ZFS snapshots, but why do I need to do that? Is LXD fixable?)
lxd process on the host is using a lot of cpu and is getting big (this after 1,200 snapshots):
PID VSTACK VSIZE RSIZE PSIZE VGROW RGROW SWAPSZ MEM CMD 1/8 10663 132K 3.7G 1.1G 0K 0K 0K 2580K 14% lxd