2

Problem

I'm trying to investigate this weird error where systemd-resolved is constantly crashing on 2 out of 4 machines. The one thing different with these two problematic machines (that I can think of), is that they are mounting a samba/cifs filesystem exported from machine #3 (on which systemd-resolved is running fine).

How do I get to the bottom of this?

OS

# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"

# uname -a
Linux prod-delayed-jobs 4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

# systemd --version
systemd 237
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid

Relevant journalctl output

Notice the /dev/null is not a device. line in the logs. This is occurring multiple times - during, before, and after systemd-resolved crashes.

Feb 12 14:23:21 prod-delayed-jobs dhclient[794]: DHCPREQUEST of [REDACTED-IP-ADDRESS] on eth0 to 172.31.1.1 port 67 (xid=0x730cc4e4)
Feb 12 14:23:21 prod-delayed-jobs dhclient[794]: DHCPACK of [REDACTED-IP-ADDRESS] from 172.31.1.1
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Stopping Network Name Resolution...
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Stopped Network Name Resolution.
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: /dev/null is not a device.
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Starting Network Name Resolution...
Feb 12 14:23:21 prod-delayed-jobs systemd[11796]: systemd-resolved.service: Failed to set up mount namespacing: Invalid argument
Feb 12 14:23:21 prod-delayed-jobs systemd[11796]: systemd-resolved.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-resolved
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=226/NAMESPACE
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Failed to start Network Name Resolution.
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: systemd-resolved.service: Service has no hold-off time, scheduling restart.
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 1.
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Stopped Network Name Resolution.
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: /dev/null is not a device.
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Starting Network Name Resolution...
Feb 12 14:23:21 prod-delayed-jobs dhclient[794]: bound to [REDACTED-IP-ADDRESS] -- renewal in 32497 seconds.
Feb 12 14:23:21 prod-delayed-jobs systemd[11813]: systemd-resolved.service: Failed to set up mount namespacing: Invalid argument
Feb 12 14:23:21 prod-delayed-jobs systemd[11813]: systemd-resolved.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-resolved
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=226/NAMESPACE
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Failed to start Network Name Resolution.
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: systemd-resolved.service: Service has no hold-off time, scheduling restart.
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 2.
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Stopped Network Name Resolution.
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: /dev/null is not a device.
Feb 12 14:23:21 prod-delayed-jobs systemd[1]: Starting Network Name Resolution...
Feb 12 14:23:22 prod-delayed-jobs systemd[11831]: systemd-resolved.service: Failed to set up mount namespacing: Invalid argument
Feb 12 14:23:22 prod-delayed-jobs systemd[11831]: systemd-resolved.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-resolved
Feb 12 14:23:22 prod-delayed-jobs systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=226/NAMESPACE
Feb 12 14:23:22 prod-delayed-jobs systemd[1]: systemd-resolved.service: Failed with result 'exit-code'.
Feb 12 14:23:22 prod-delayed-jobs systemd[1]: Failed to start Network Name Resolution.
Feb 12 14:23:22 prod-delayed-jobs systemd[1]: systemd-resolved.service: Service has no hold-off time, scheduling restart.
Feb 12 14:23:22 prod-delayed-jobs systemd[1]: systemd-resolved.service: Scheduled restart job, restart counter is at 3.

/etc/fstab contents - in case this is relevant

UUID=2f54e8e6-ff9c-497a-88ea-ce159f6cd283 /               ext4    discard,errors=remount-ro 0       1
/dev/fd0        /media/floppy0  auto    rw,user,noauto,exec,utf8 0       0
//[REDACTED]/uploaded_files /home/myuser/shared cifs rw,username=myuser,password=[REDACTED],uid=myuser,gid=myuser 0 0

Few systemd configuration files are pointing to /dev/null

This is the case on all the 4 machines, however only 2 out of these 4 are having trouble with systemd-resolved. Strangely, hostname.service is pointing to /dev/null - wonder why?

lrwxrwxrwx 1 root root    9 Nov 15 21:45 bootlogd.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 bootlogs.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 bootmisc.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 checkfs.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 checkroot-bootclean.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 checkroot.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 cryptdisks-early.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 cryptdisks.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 fuse.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 halt.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 hostname.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 hwclock.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 killprocs.service -> /dev/null
lrwxrwxrwx 1 root root    9 Apr 12  2018 lvm2.service -> /dev/null
lrwxrwxrwx 1 root root    9 Jul  2  2018 mdadm.service -> /dev/null
lrwxrwxrwx 1 root root    9 Jul  2  2018 mdadm-waitidle.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 motd.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 mountall-bootclean.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 mountall.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 mountdevsubfs.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 mountkernfs.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 mountnfs-bootclean.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 mountnfs.service -> /dev/null
lrwxrwxrwx 1 root root    9 Jan 16  2018 nfs-common.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 rc.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 rcS.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 reboot.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 rmnologin.service -> /dev/null
lrwxrwxrwx 1 root root    9 Dec  6 11:10 screen-cleanup.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 sendsigs.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 single.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 stop-bootlogd.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 stop-bootlogd-single.service -> /dev/null
lrwxrwxrwx 1 root root    9 Dec  6 11:07 sudo.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 umountfs.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 umountnfs.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 umountroot.service -> /dev/null
lrwxrwxrwx 1 root root    9 Nov 15 21:45 x11-common.service -> /dev/null

/lib/systemd/system

[Unit]
Description=Network Name Resolution
Documentation=man:systemd-resolved.service(8)
Documentation=https://www.freedesktop.org/wiki/Software/systemd/resolved
Documentation=https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
Documentation=https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
DefaultDependencies=no
After=systemd-sysusers.service systemd-networkd.service
Before=network.target nss-lookup.target shutdown.target
Conflicts=shutdown.target
Wants=nss-lookup.target

[Service]
Type=notify
Restart=always
RestartSec=0
ExecStart=!!/lib/systemd/systemd-resolved
WatchdogSec=3min
User=systemd-resolve
CapabilityBoundingSet=CAP_SETPCAP CAP_NET_RAW CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_SETPCAP CAP_NET_RAW CAP_NET_BIND_SERVICE
PrivateTmp=yes
PrivateDevices=yes
ProtectSystem=strict
ProtectHome=yes
ProtectControlGroups=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
MemoryDenyWriteExecute=yes
RestrictRealtime=yes
RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_INET AF_INET6
SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module @mount @obsolete @raw-io @reboot @swap

1 Answer 1

0

I had the same problem today with systemd-resolved that works flawlessly before, and I solved it by setting ProtectSystem=full with systemctl edit systemd-resolved.service

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.