LXD ZFS snapshot creation slows down as number of snapshots increases

Question

Ubuntu 18.04 LTS - LXD Ubuntu launch using loopback ZFS (about 400MB)

# lxc launch ubuntu:16.04 test -s ianzfspool

(I have one other idle multi-GB container using the same ianzfspool.) I have lots of space in the pool:

# zpool list                                    
NAME         SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
ianzfspool  99.5G  13.7G  85.8G         -     2%    13%  1.00x  ONLINE  -

If I loop, using lxc to create snapshots of the idle test container (400MB), the first ones create in a few seconds. As the number of lxc snapshots grows (currently over 1,200), now they take minutes to create. I have a script running in the test container that puts the current date in a file in /tmp every few seconds, so I have one file that is changing in the container in every snapshot; otherwise, the container is mostly idle. The first lxc snapshots used about 25K; the current ones (after 1,200 snapshots) use 66K. (After 2,470 snapshots they each use 115K and after 7,500 snapshots they each use 293K.)

#!/bin/sh -u                                                                   
# Create snapshots of the Ubuntu test container.
count=0
while : ; do
    count=$(( count + 1 ))
    cp=$( printf "%05d" $count )
    lxc snapshot test snap$cp
    echo "$0: done $cp" >/tmp/icount.txt
done

---

Edit 1: I currently am up to about 2,470 lxc snapshots and each new snapshot is taking about four minutes to create and uses about 115K. If I call zfs snapshot to take a snapshot of the container directly, instead of using lxc snapshot, the snapshot takes less than a second, even with 2,470 existing snapshots. The direct zfs snapshot only uses 20K (instead of 115K). Running zfs list takes only a second or two. If I run lxc list (instead of zfs list), that currently takes over four minutes with 2,470 existing snapshots. So the snapshot creation and listing slowdown isn't with ZFS, it's with LXD. Indeed, the lxd process itself has used 4,253 CPU seconds and has a VSIZE of 3.9G so far in this snapshot creation experiment.

I paused the lxc snapshot creation script and wrote one to loop creating direct zfs snapshots, and it has created more snapshots in five minutes than lxc has in three days. I paused it after about 3,000 direct ZFS snapshots.

I just re-ran lxc list and of course it still only shows the 2,471 lxc snapshots, but now it ran in only 40 seconds instead of four minutes. I just created another lxc snapshot and it only took 49 seconds instead of four minutes. What changed? Has creating a bunch of direct zfs snapshots somehow sped up lxc snapshot creation and listing? The lxc snapshots are still way slower to create than direct zfs snapshots, but something has improved with lxc snapshot creation (from four minutes to 49 seconds).

I let the ZFS snapshot creation proceed until 10,000 direct ZFS snapshots were created. (So now I have 2,472 lxc snapshots and 10,000 direct zfs snapshots of this test container.) The lxc list now takes 90 seconds (instead of 40) and lxc snapshot takes 100 seconds (instead of 49). Using ZFS directly is still two orders of magnitude faster than going via LXD.

---

Edit 2: After a reboot, LXD snapshot creation has sped up. I'm at 7,470 snapshots created via my lxc snapshot loop and each snapshot is now taking about 30 seconds to create and has a size of 293K. The lxc list takes 45 seconds. The zfs list takes 105 seconds and produces 17,507 lines of output (which includes the 10,000 direct ZFS snapshots). Doing zfs snapshot ianzfspool/containers/test@iansnap10001 directly (not via LXD) takes less than half a second -- still much faster than via LXD.

---

Where can I find documentation on why LXD snapshot creation is slowing down, and how I might speed things up? (If LXD with ZFS snapshots were as cheap as I was led to believe, I was hoping to run snapshots every 5 minutes and keep them for about a week, which would be 2,016 snapshots. I've read that LVM snapshots are a performance penalty, but nothing I've read said the same about LXD with ZFS. I see that I could bypass LXD and use ZFS snapshots, but why do I need to do that? Is LXD fixable?)

The lxd process on the host is using a lot of cpu and is getting big (this after 1,200 snapshots):

  PID VSTACK   VSIZE   RSIZE  PSIZE   VGROW   RGROW SWAPSZ   MEM  CMD        1/8
10663   132K    3.7G    1.1G     0K      0K      0K  2580K   14%  lxd

Answer 1

I don’t have experience with lxd snapshot, so the best I can do is guess about its implementation and tell you more details about ZFS snapshots.

ZFS snapshots are designed so that they do not add any overhead to ongoing reads and writes. (As you mentioned, LVM snapshots do add a performance penalty, in the form of write amplification.) To achieve this, ZFS has an internal operation called a txg_sync which is like a batch write of all recent async IOs, and it happens automatically every 10 seconds, as well as any time the file system structure changes (such as when you take a snapshot). This causes a bunch of IOs to happen, so in theory can cause a slowdown of concurrent synchronous writes due to congestion (although sync writes should get priority). However, it seems you have written almost nothing to your file system (and presumably not a lot more to the rest of the pool).

My guess is that it is actually metadata reads which are getting slow. In theory, lxd snapshot could just take a ZFS snapshot and keep going, which should take a constant (assuming constant IO load) amount of time. However, I’m guessing that it also tries to list all snapshots during that operation (equivalent to zfs list), and that involves reading a whole bunch of metadata for every snapshot, which is often spread all over the pool, and grows linearly with the number of snapshots. To check if that’s the case, you could try timing a zfs snapshot on the filesystem directly, and doing a zfs list -t all on the file system and its snapshots, and see which one takes way way longer than the other (should be the second one).

In the ZFS community this is something of a known problem, but it’s hard to fix because it would involve changing a lot of on-disk metadata structures to improve, and keeping this many snapshots is pretty unusual. If lxd tried harder to avoid listing all the snapshots to run its snapshot operation, I think that could solve your issue.

Alternately, you could take snapshots every 5 minutes, but then delete them relatively quickly and only keep (for example) 5m, 10m, 15m, 30m, 1h, 2h, 3h, 6h, 12h, 24h, etc. snapshots. That gives you high resolution during the window when it’s most likely you would need it, and also wouldn’t cause lxd snapshot to slow down.