1

I've set up a few containers on Xenial server, using a ZFS dataset for storage (I'm not sure if that's relevant; we'll get to that). They were all working fine.

Trying to restart one of them, mail-server I now get

error: Error calling 'lxd forkstart mail-server /var/lib/lxd/containers /var/log/lxd/mail-server/lxc.conf': err='exit status 1'
Try `lxc info --show-log mail-server` for more info

and looking at the log I see

        lxc 20160518112750.405 ERROR    lxc_conf - conf.c:mount_rootfs:807 - No such file or directory - failed to get real path for '/var/lib/lxd/containers/mail-server/rootfs'
        lxc 20160518112750.405 ERROR    lxc_conf - conf.c:setup_rootfs:1221 - failed to mount rootfs
        lxc 20160518112750.405 ERROR    lxc_conf - conf.c:do_rootfs_setup:3611 - failed to setup rootfs for 'mail-server'

This isn't surprising, since when I check

$ sudo ls -l /var/lib/lxd/containers/mail-server.zfs
total 0

However, I get the same if I check another container:

$ sudo ls -l /var/lib/lxd/containers/smb-server.zfs
total 0

but that container is currently running:

$ lxc list
+---------------+---------+--------------------------------+------+------------+-----------+
|     NAME      |  STATE  |              IPV4              | IPV6 |    TYPE    | SNAPSHOTS |
+---------------+---------+--------------------------------+------+------------+-----------+
| smb-server    | RUNNING | 192.168.2.11 (eth0)            |      | PERSISTENT | 2         |
+---------------+---------+--------------------------------+------+------------+-----------+
| mail-server   | STOPPED |                                |      | PERSISTENT | 0         |
+---------------+---------+--------------------------------+------+------------+-----------+
| ub-test       | STOPPED |                                |      | PERSISTENT | 0         |
+---------------+---------+--------------------------------+------+------------+-----------+
| www-server    | STOPPED |                                |      | PERSISTENT | 1         |
+---------------+---------+--------------------------------+------+------------+-----------+

$ lxc exec smb-server -- /bin/bash
root@smb-server:~# ls -l /
total 88
drwxr-xr-x   2 root   root    173 May 17 10:10 bin
drwxr-xr-x   3 root   root      3 Apr 20 17:58 boot
drwxr-xr-x   9 root   root    480 May 17 21:40 dev
...

www-server was also running, and showning nothing in ls, and once I stopped it I can no longer start it again, for the same reason. The only container whose rootfs I can still find, ub-test:

$ sudo ls -l /var/lib/lxd/containers/ub-test.zfs
total 22
-rw-r--r--  1 lxd    messagebus 1425 Apr 20 18:45 metadata.yaml
drwxr-xr-x 22 100000     100000   22 Apr 22 11:38 rootfs
drwxr-xr-x  2 lxd    messagebus    7 Apr 20 18:45 templates

I can indeed stop and start at will.


My only guess as to the reason for this change (the only thing I can recall doing that might conceivably affect this issue) is that I deleted another container yesterday, ub-test2. I can still see a trace of that with:

$ sudo zfs list
NAME                                                                                            USED  AVAIL  REFER  MOUNTPOINT
tank                                                                                      2.80T  2.44T    96K  /tank
tank/lxd                                                                                  4.71G  2.44T    96K  none
tank/lxd/containers                                                                       3.34G  2.44T    96K  none
tank/lxd/containers/smb-server                                                         2.31G  2.44T  2.59G  /var/lib/lxd/containers/smb-server.zfs
tank/lxd/containers/mail-server                                                            279M  2.44T   746M  /var/lib/lxd/containers/mail-server.zfs
tank/lxd/containers/ub-test                                                                485M  2.44T   952M  /var/lib/lxd/containers/ub-test.zfs
tank/lxd/containers/www-server                                                             289M  2.44T   757M  /var/lib/lxd/containers/www-server.zfs
tank/lxd/deleted                                                                           702M  2.44T    96K  none
tank/lxd/deleted/images                                                                    702M  2.44T    96K  none
tank/lxd/deleted/images/f4c4c60a6b752a381288ae72a1689a9da00f8e03b732c8d1b8a8fcd1a8890800   702M  2.44T   702M  none
tank/lxd/images                                                                            704M  2.44T    96K  none
tank/lxd/images/d23ee1f4fd284aeaba6adeb67cccf7b871e96178d637fec96320aab7cc9634b1           704M  2.44T   704M  /var/lib/lxd/images/d23ee1f4fd284aeaba6adeb67cccf7b871e96178d637fec96320aab7cc9634b1.zfs

You can also see there that /var/lib/lxd/containers/mail-server.zfs (and smb-server, www-server) are still using storage (~280M), even though ls returns nothing, which makes me suspect this issue is somehow related to ZFS mounts gone awry; my first move would've been to restart the whole system and see if that clears things up, but I fear that would cause the last running container to die as well, making it harder to troubleshoot the issue.


For reference, here's my containers' storage directory:

$ sudo ls -l /var/lib/lxd/containers
total 37
lrwxrwxrwx 1 root   root      41 Apr 25 09:19 smb-server -> /var/lib/lxd/containers/deluge-ub1604.zfs
drwxr-xr-x 2 root   root    4096 Apr 25 09:19 smb-server.zfs
-rw-r--r-- 1 root   root   19011 May 18 11:46 lxc-monitord.log
lrwxrwxrwx 1 root   root      39 May 18 02:02 mail-server -> /var/lib/lxd/containers/mail-server.zfs
drwxr-xr-x 2 100000 100000  4096 May 18 02:02 mail-server.zfs
lrwxrwxrwx 1 root   root      35 Apr 22 11:38 ub-test -> /var/lib/lxd/containers/ub-test.zfs
drwxr-xr-x 4 100000 100000     5 Apr 22 11:37 ub-test.zfs
lrwxrwxrwx 1 root   root      38 May 17 09:31 www-server -> /var/lib/lxd/containers/smb-server.zfs
drwxr-xr-x 2 100000 100000  4096 May 17 09:31 www-server.zfs

3 Answers 3

3

I would like to tell you that I got the same issue with all my Xenial container servers (5).

To add more information to our issue, the mounts do not appear anymore when you type mount.

If you type zfs mount -a it only mounts back the non-running containers, with running containers you get the following error: "filesystem 'lxd/containers/container_name' is already mounted"

So to fix the issue completely you need to stop the container(s) first and then issue zfs mount -a.

To conclude either there is some timeout on these zfs mounts or some cron job did remove them somehow, but this is definitely a bug.

Hopefully this didn't damage any container...

2
  • Yes, except that without finding any indication of what went wrong (I struck out looking in /var/log/lxd/<container-name>) we can't report this bug. If you do find further proof / information, please let me know (you can always comment on your own answer. Just mention me like @JonathanY.) May 21, 2016 at 13:16
  • 1
    I just hit the same in Xenial LXD and also still in Xenial backports which atm is 2.7-0ubuntu2. The log is slightly longer and might help so look at pastebin.com/raw/CbMwJZBF - fortunately the workaround with stop/delete/zfs mount worked as well. Well but OTOH it seems I can reproduce the issue - going to create a bug report for it. Jan 24, 2017 at 7:35
1

I still can't imagine what went wrong, but running sudo zfs mount -a fixed the issue (as mentioned above, the ZFS datasets did exist--as evident by their use of storage--they were just unmounted for some reason) for every container not running.

Stopping www-server, I could fix its mountpoint as well with the same command, then start it again.

N.B. If anyone can explain the cause behind this error in the next couple of days, I'll accept their answer. Otherwise, I'll leave this answer here for anyone else who experiences the same issue (although, having running containers was all that stopped me from rebooting, which would've fixed the issue too--since all ZFS datasets are mounted on startup--so this issue is far from critical).

2
  • Did you inspect the mount attributes of each container's dataset, working and non-working? sudo zfs get all <dataset>|grep mount. Maybe it yields another clue. May 18, 2016 at 13:18
  • @andreas I can't glean anything from that now that I've remounted all my containers with sudo zfs mount -a, except to comment that they were all set appropriate mountpoints before, that somehow got unmounted. Now they are mounted again, of course, with the proper paths (in var/lib/lxd/containers), which is what zfs get all shows me. May 18, 2016 at 13:23
0

Saw this today on a fresh Xenial install, also after having deleted some containers the day before.

Removing the /var/lib/lxd/containername.zfs mount point (rm) and then running zfs mount -a fixed it

Quite disconcerting that you can't delete a container without worrying about mount points getting screwed up!

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .