1

Short version

When booting exactly the same system repeatedly, sometimes the boot messages start with

[  ...] systemd[1]: Set hostname to <liab>.
[  ...] systemd[1]: Mounted /.
[  ...] systemd[1]: Mounted /host.
[  ...] systemd[1]: Cannot add dependency job for unit display-manager.service, ignoring: Unit display-manager.service failed to load: No such file or directory.

while at other times they start with

[  ...] systemd[1]: Set hostname to <liab>.
[  ...] systemd[1]: Unit host.mount is bound to inactive unit dev-sda1.device. Stopping, too.
[  ...] systemd[1]: Cannot add dependency job for unit display-manager.service, ignoring: Unit display-manager.service failed to load: No such file or directory.

and lack several service startup messages later on, compared to the former case. How can I find the cause for this?

Long version

I've got a very non-standard setup here, since I'm building a kiosk setup.

Creation of the root file system

I didn't runn a full Ubuntu installer, but start off with debootstrap --variant=minbase … vivid, adding some packages and customizing stuff in a chroot. That chroot is then used to create a squashfs which serves as the root file system of my setup.

Mounting from initramfs

That read-only squashfs is contained in a file on some host file system, which might be vfat or ext4 or whatever. It gets mounted by the initramfs. To achieve this I have a file in /etc/initramfs-tools/conf.d/ which sets LOOP to the path of the squashfs in the root file system, and also sets LOOPFSTYPE=squashfs. The squashfs contains an empty directory called /host which causes the /usr/share/initramfs-tools/scripts/local to mount -o move the host file system to that directory. Therefore, when the initramfs hands control over to systemd, I'll have / and /host mounted and ready for use.

fstab

My /etc/fstab reads

# <fs>          <mountpoint>    <type>          <opts>                                          <dump/pass>
/dev/loop0      /               squashfs        nodev,noauto,ro                                 0 0
/host/boot      /boot           auto            bind,ro                                         0 0
none            /run            tmpfs           noexec,nodev,nosuid                             0 0
none            /tmp            tmpfs           noexec,nodev,nosuid                             0 0
none            /var/tmp        tmpfs           noexec,nodev,nosuid                             0 0
none            /var/log        tmpfs           noexec,nodev,nosuid,size=5%                     0 0
none            /home/liab      tmpfs           nodev,nosuid,uid=liab,gid=liab,mode=0770        0 0

So I have an entry for / but it's marked noauto since the initramfs already took care of this, right? Nevertheless, the options listed there apparently get applied at some point, which I noticed while I accidentially had nosuid in there. I don't have an entry for /host since I don't know the device I'd have to use for that. The initramfs can work out that device from the boot loader configuration, but I wouldn't want to include any file system id or some such it into the squashfs, since the same squashfs should be usable from different host file systems.

Booting

Currently, I'm only booting this setup in qemu. And I'm using its -snapshot option, with a file system my user account can't modify. So I'm certain that there is no change at all to the underlying file system image, and any difference in behavior must therefore be the consequence of some non-determinism, most likely some kind of race condition. My qemu invocation currently reads somewhat like this:

qemu-system-x86_64 \
-serial stdio \
-kernel …/boot/vmlinuz-3.19.0-15-generic \
-append 'console=ttyS0 root=/dev/sda1' \
-initrd …/initrd.img-3.19.0-15-generic \
-snapshot -hda …/hdimg

So I'll get a screen which will eventually show a GUI, but I'll also get boot messages printed to the terminal where I started qemu, which the guest kernel will see as a serial console. It's those messages to the console which have me worried.

Apparently, the initramfs does it's job just fine, and then hands control over to systemd. Systemd in turn will display a greeting, set the hostname, and then apparently follow one of two possible paths.

The first path causes mount messages for / and /host. Starting with a message [ OK ] Reached target Swap., many of the boot steps will be inidcated by a message line starting in [ OK ] with the OK printed in green. Later on, probably in response to Starting Journal Service, the dmesg-like timestamped messages disappear, presumably redirected to some logging daemon, and only those [ OK ] messages remain.

The second path does not mention remounting / and /host early on. Instead it writes Unit host.mount is bound to inactive unit dev-sda1.device. Stopping, too.. The dev-sda1.device in there corresponds to the root=/dev/sda1 kernel argument; changing that to some file system ID changes the error message accordingly. There never are any colored lines in this boot process. The corresponding lines simply seem to be missing altogether, even if the timestamped lines log start and completion of many of the same steps. Order of these steps differs. Which means that once the journal service got started, I don't see any messages at all any more.

So far, I can't tell what other actual differences these different bootups cause for the running system. But I don't like non-deterministic behavior which I don't understand.

Debugging

In an attempt to debug this situation, I added systemd.log_target=console systemd.log_level=debug to the kernel command line. One undesired effect of this was that initramfs execution would print a million lines apparently during udev event processing. The relevant portion of the log appears to be the following. If I get the green OK likes, it reads

Using notification socket /run/systemd/notify
Successfully created private D-Bus server.
dev-loop0.device changed dead -> tentative
dev-sda1.device changed dead -> tentative
Trying to enqueue job host.mount/start/fail
Installed new job host.mount/start as 1
Installed new job -.slice/start as 4
Installed new job system.slice/start as 3
Installed new job dev-sda1.device/start as 5
Installed new job -.mount/start as 2
Enqueued job host.mount/start as 1
host.mount changed dead -> mounted
Job host.mount/start finished, result=done
-.mount changed dead -> mounted
Job -.mount/start finished, result=done
Activating default unit: default.target
Failed to load configuration for all.target: No such file or directory
Failed to load configuration for bluetooth.service: No such file or directory
Failed to load configuration for hal.service: No such file or directory
Failed to load configuration for kbd.service: No such file or directory
Failed to load configuration for console-screen.service: No such file or directory
Failed to load configuration for plymouth-quit-wait.service: No such file or directory
Failed to load configuration for plymouth-quit.service: No such file or directory
Failed to load configuration for display-manager.service: No such file or directory
Trying to enqueue job graphical.target/start/isolate
Cannot add dependency job for unit display-manager.service, ignoring: Unit display-manager.service failed to load: No such file or directory.
Job dev-sda1.device/start finished, result=canceled
Installed new job syslog.socket/start as 90

and if I don't get those lines it instead reads

Using notification socket /run/systemd/notify
Successfully created private D-Bus server.
host.mount changed dead -> mounted
Unit host.mount is bound to inactive unit dev-sda1.device. Stopping, too.
Trying to enqueue job host.mount/stop/fail
Installed new job host.mount/stop as 1
Enqueued job host.mount/stop as 1
-.mount changed dead -> mounted
dev-loop0.device changed dead -> tentative
dev-sda1.device changed dead -> tentative
Activating default unit: default.target
Failed to load configuration for bluetooth.service: No such file or directory
Failed to load configuration for hal.service: No such file or directory
Failed to load configuration for kbd.service: No such file or directory
Failed to load configuration for console-screen.service: No such file or directory
Failed to load configuration for all.target: No such file or directory
Failed to load configuration for plymouth-quit-wait.service: No such file or directory
Failed to load configuration for plymouth-quit.service: No such file or directory
Failed to load configuration for display-manager.service: No such file or directory
Trying to enqueue job graphical.target/start/isolate
Cannot add dependency job for unit display-manager.service, ignoring: Unit display-manager.service failed to load: No such file or directory.
Job host.mount/stop finished, result=canceled
Startup finished in 13.247s (kernel) + 2.322s (userspace) = 15.569s.
Installed new job systemd-modules-load.service/start as 48

Not sure whether this is of any use.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.