I've set up a few containers on Xenial server, using a ZFS dataset for storage (I'm not sure if that's relevant; we'll get to that). They were all working fine.
Trying to restart one of them, mail-server
I now get
error: Error calling 'lxd forkstart mail-server /var/lib/lxd/containers /var/log/lxd/mail-server/lxc.conf': err='exit status 1'
Try `lxc info --show-log mail-server` for more info
and looking at the log I see
lxc 20160518112750.405 ERROR lxc_conf - conf.c:mount_rootfs:807 - No such file or directory - failed to get real path for '/var/lib/lxd/containers/mail-server/rootfs'
lxc 20160518112750.405 ERROR lxc_conf - conf.c:setup_rootfs:1221 - failed to mount rootfs
lxc 20160518112750.405 ERROR lxc_conf - conf.c:do_rootfs_setup:3611 - failed to setup rootfs for 'mail-server'
This isn't surprising, since when I check
$ sudo ls -l /var/lib/lxd/containers/mail-server.zfs
total 0
However, I get the same if I check another container:
$ sudo ls -l /var/lib/lxd/containers/smb-server.zfs
total 0
but that container is currently running:
$ lxc list
+---------------+---------+--------------------------------+------+------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+---------------+---------+--------------------------------+------+------------+-----------+
| smb-server | RUNNING | 192.168.2.11 (eth0) | | PERSISTENT | 2 |
+---------------+---------+--------------------------------+------+------------+-----------+
| mail-server | STOPPED | | | PERSISTENT | 0 |
+---------------+---------+--------------------------------+------+------------+-----------+
| ub-test | STOPPED | | | PERSISTENT | 0 |
+---------------+---------+--------------------------------+------+------------+-----------+
| www-server | STOPPED | | | PERSISTENT | 1 |
+---------------+---------+--------------------------------+------+------------+-----------+
$ lxc exec smb-server -- /bin/bash
root@smb-server:~# ls -l /
total 88
drwxr-xr-x 2 root root 173 May 17 10:10 bin
drwxr-xr-x 3 root root 3 Apr 20 17:58 boot
drwxr-xr-x 9 root root 480 May 17 21:40 dev
...
www-server
was also running, and showning nothing in ls
, and once I stopped it I can no longer start it again, for the same reason. The only container whose rootfs
I can still find, ub-test
:
$ sudo ls -l /var/lib/lxd/containers/ub-test.zfs
total 22
-rw-r--r-- 1 lxd messagebus 1425 Apr 20 18:45 metadata.yaml
drwxr-xr-x 22 100000 100000 22 Apr 22 11:38 rootfs
drwxr-xr-x 2 lxd messagebus 7 Apr 20 18:45 templates
I can indeed stop and start at will.
My only guess as to the reason for this change (the only thing I can recall doing that might conceivably affect this issue) is that I deleted another container yesterday, ub-test2
. I can still see a trace of that with:
$ sudo zfs list
NAME USED AVAIL REFER MOUNTPOINT
tank 2.80T 2.44T 96K /tank
tank/lxd 4.71G 2.44T 96K none
tank/lxd/containers 3.34G 2.44T 96K none
tank/lxd/containers/smb-server 2.31G 2.44T 2.59G /var/lib/lxd/containers/smb-server.zfs
tank/lxd/containers/mail-server 279M 2.44T 746M /var/lib/lxd/containers/mail-server.zfs
tank/lxd/containers/ub-test 485M 2.44T 952M /var/lib/lxd/containers/ub-test.zfs
tank/lxd/containers/www-server 289M 2.44T 757M /var/lib/lxd/containers/www-server.zfs
tank/lxd/deleted 702M 2.44T 96K none
tank/lxd/deleted/images 702M 2.44T 96K none
tank/lxd/deleted/images/f4c4c60a6b752a381288ae72a1689a9da00f8e03b732c8d1b8a8fcd1a8890800 702M 2.44T 702M none
tank/lxd/images 704M 2.44T 96K none
tank/lxd/images/d23ee1f4fd284aeaba6adeb67cccf7b871e96178d637fec96320aab7cc9634b1 704M 2.44T 704M /var/lib/lxd/images/d23ee1f4fd284aeaba6adeb67cccf7b871e96178d637fec96320aab7cc9634b1.zfs
You can also see there that /var/lib/lxd/containers/mail-server.zfs
(and smb-server
, www-server
) are still using storage (~280M), even though ls
returns nothing, which makes me suspect this issue is somehow related to ZFS mounts gone awry; my first move would've been to restart the whole system and see if that clears things up, but I fear that would cause the last running container to die as well, making it harder to troubleshoot the issue.
For reference, here's my containers' storage directory:
$ sudo ls -l /var/lib/lxd/containers
total 37
lrwxrwxrwx 1 root root 41 Apr 25 09:19 smb-server -> /var/lib/lxd/containers/deluge-ub1604.zfs
drwxr-xr-x 2 root root 4096 Apr 25 09:19 smb-server.zfs
-rw-r--r-- 1 root root 19011 May 18 11:46 lxc-monitord.log
lrwxrwxrwx 1 root root 39 May 18 02:02 mail-server -> /var/lib/lxd/containers/mail-server.zfs
drwxr-xr-x 2 100000 100000 4096 May 18 02:02 mail-server.zfs
lrwxrwxrwx 1 root root 35 Apr 22 11:38 ub-test -> /var/lib/lxd/containers/ub-test.zfs
drwxr-xr-x 4 100000 100000 5 Apr 22 11:37 ub-test.zfs
lrwxrwxrwx 1 root root 38 May 17 09:31 www-server -> /var/lib/lxd/containers/smb-server.zfs
drwxr-xr-x 2 100000 100000 4096 May 17 09:31 www-server.zfs