systemd starts dnsmasq before starting network manager?

Question

Looks to me that my systemd starts dnsmasq before starting network manager. How can I confirm this? How can I force systemd starts dnsmasq after starting network manager?

Reason behind this is, as explained in NetworkManager not updating /var/run/networkmanager/resolv.conf,

• I'm using dnsmasq as my LAN DNS server, and
• I defined my two IP addresses in NetworkManager.

When my system started my dnsmasq is not working, and I found that the reason is it is not listening to my two IP addresses at all. Only after restarting it, it started to listen to my two IP addresses:

$ lsof -i tcp:53
COMMAND   PID            USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
systemd-r 544 systemd-resolve   13u  IPv4  18667      0t0  TCP 127.0.0.53:domain (LISTEN)
dnsmasq   793         dnsmasq    5u  IPv4  23719      0t0  TCP localhost.my.box.name:domain (LISTEN)
dnsmasq   793         dnsmasq    7u  IPv6  23721      0t0  TCP ip6-localhost:domain (LISTEN)

/etc/init.d/dnsmasq restart
[ ok ] Restarting dnsmasq (via systemctl): dnsmasq.service.

$ lsof -i tcp:53
COMMAND     PID            USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
systemd-r   544 systemd-resolve   13u  IPv4  18667      0t0  TCP 127.0.0.53:domain (LISTEN)
dnsmasq   17895         dnsmasq    5u  IPv4 601091      0t0  TCP 192.168.0.10:domain (LISTEN)
dnsmasq   17895         dnsmasq    7u  IPv4 601093      0t0  TCP 192.168.0.11:domain (LISTEN)
dnsmasq   17895         dnsmasq    9u  IPv4 601095      0t0  TCP localhost...

$ lsb_release -a 
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

# my systemd is not the latest but I think that shouldn't matter:
$ apt-cache policy systemd
systemd:
  Installed: 237-3ubuntu10
  Candidate: 237-3ubuntu10.9
  Version table:
     237-3ubuntu10.9 500
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
 *** 237-3ubuntu10 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
        100 /var/lib/dpkg/status

Update:

@heynnema, Thanks for the systemd-analyze --no-pager blame, I've confirmed that the start order is as expected

NetworkManager -> systemd-resolved -> dnsmasq.

So indeed, it is about dnsmasq and systemd-resolved stepping on each others toes. I've added DNSStubListener=no since. I have two following up questions now.

• Why it is OK for dnsmasq restart to work just fine? Its working fine gave me the impression that it is OK to have them both working.
• Actually, I used systemctl stop systemd-resolved and systemctl disable systemd-resolved to completely disable systemd-resolved, instead of just stopping its listener previously. I don't know why it does not persist over reboot.

Answer 1

I had the same issue, but only on 16.04 systems updated to 18.04. I am also using dnsmasq as my LAN DNS and DHCP server.
On a freshly installed 18.04, it is working out of the box, without having to modify /etc/systemd/resolved.conf (DNSStubListener).

Answer 2

Unresolved bug in Ubuntu 18.04. I'll show you how I worked around it. Maybe others have a more elegant solution, but I'll start the ball rolling with my own.

The problem lives in the [Unit] section in the After= directive which must be modified as follows to raise dnsmasq.service unbroken on boot:

sed -i 's/After=network.target/After=NetworkManager-wait-online.service/' /lib/systemd/system/dnsmasq.service

When testing I initially tried After=NetworkManager.service which seemed a sensible choice. But after a reboot dnsmasq was still rising-up broken on boot. So I did:

systemd-analyze critical-chain

And found NetworkManager-wait-online.service started even later. That's how I found the appropriate relative order in which to raise dnsmasq.service. HTH you if you've stumbled onto this issue-