20

The systemd's DNS lovated at 127.0.0.53 appears to be working except when I query for local machines by name. But if I query for them and specifically specify the local DNS server (my router) then I get the proper reply. But the config file says it is also using the router as the search address. Any thoughts?

I am running Ubuntu 18.04 on my Dell laptop.

Incorrect Results:

$ nslookup web1

Server:     127.0.0.53
Address:    127.0.0.53#53

** server can't find web1: SERVFAIL

Also Fails

$ nslookup -i wlp3s0 web1
nslookup: couldn't get address for 'web1': not found

Correct Results:

$ nslookup web1 192.168.1.1

Server:     192.168.1.1
Address:    192.168.1.1#53

Name:   web1
Address: 192.168.1.107

Configuration Info systemd-resolve

$ systemd-resolve --status

Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 3 (wlp3s0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.1.1
          DNS Domain: wp.comcast.net

Link 2 (enp2s0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Configuration Info NetworkManager

$ cat /etc/NetworkManager/NetworkManager.conf
[main]
plugins=ifupdown,keyfile

[ifupdown]
managed=false

[device]
wifi.scan-rand-mac-address=no

So how do I get nslookup to return the correct answer? Link 3 appears to be the correct information (my wifi connection) and my DNS on the router is returning the correct answer but the local cache never tries to look up the address (or so it seems).

6
  • Check out this post and see if that solve your problem: askubuntu.com/questions/1034064/…
    – user822833
    May 26 '18 at 13:32
  • I don't have dns=dnsmasq in my config file. I am updating my question to show this.
    – schworak
    May 26 '18 at 13:52
  • Which version of Ubuntu are you running and can you also update your post with the IP configuration?
    – user822833
    May 26 '18 at 14:13
  • I am running Ubuntu 18.04 on a Dell laptop.
    – schworak
    May 28 '18 at 12:32
  • could you plz try nslookup -i wlp3s0 web1
    – cmak.fr
    May 28 '18 at 16:05
25

I found the fix that worked for me.

my resolv.conf file was pointing to the wrong place. This seems like a bug in Ubuntu as it happened on my laptop (the machine I first noticed this issue on) and on a fresh install of Ubuntu 18.04 Server.

The Default

$ ls -l /etc/resolv.conf

lrwxrwxrwx 1 root root 39 Apr 26 12:07 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

I deleted this and pointed to the correct file. After rebooting, this solved my issue. And I was even able to switch networks on my laptop and the DNS switched correctly. Of course when on external networks I can't resolve any of my local machines but that is expected. As soon as I switch back to my local network, all the local machines resolve correctly because my router is the DNS.

The Fix

$ sudo unlink /etc/resolv.conf
$ sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf
$ ls -l /etc/resolv.conf

lrwxrwxrwx 1 root root 32 May 29 08:48 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf

$ sudo reboot

After that, everything worked as I expected and 127.0.0.53 is no longer being used at all.

The Correct Results

$ nslookup web1

Server:     192.168.1.1
Address:    192.168.1.1#53

Name:   web1
Address: 192.168.1.107

$ nslookup google.com

Server:     192.168.1.1
Address:    192.168.1.1#53

Non-authoritative answer:
Name:   google.com
Address: 172.217.7.174
Name:   google.com
Address: 2607:f8b0:4004:80e::200e
9
  • Please report this bug using ubuntu-bug resolvconf. May 29 '18 at 17:57
  • When I start to send it says resolvconf (not installed). Is resolvconf and systemd-resolve the same thing?
    – schworak
    May 31 '18 at 0:28
  • 1
    I had the same problem and this solved it. I didn't reboot the laptop only restarted the network with sudo /etc/init.d/networking restart
    – matteol
    Nov 3 '18 at 9:02
  • 1
    Wow, this is crazy. Thank you. Did this bug ever get fixed? Is it a Docker-specific bug? I think the resolv.conf is set up this way for docker bridge network DNS right? Dec 18 '18 at 0:25
  • 1
    just raised it as I could not find it: bugs.launchpad.net/ubuntu/+source/systemd/+bug/1872015 Apr 10 '20 at 7:19
13

Your resolv.conf file was not pointing to the wrong place — ../run/systemd/resolve/stub-resolv.conf is where it's supposed to point at by default.

The problem is that systemd-resolved does not pass non-dotted names onto DNS. Apparently this is working "as designed". See this github issue which states that "resolved will never allow single-label lookups to leak onto unicast DNS".

Whether or not you agree with the reasoning in that github issue, there is a way to fix this. It doesn't even require making any changes to the default setup on your Ubuntu machine:

  1. First, your LAN's DNS must have a domain name.

    If you're using dnsmasq, add the following to /etc/dnsmasq.conf on your DNS server:

    expand-hosts
    domain=your-domain # replace "your-domain" with domain of your choice
    

    You should now be able to resolve LAN hostnames if your add the domain:

    nslookup web1.your-domain
    
  2. Second, make sure the name for your LAN's domain is also set in your DHCP server if it's different from your DNS server. On my DHCP server (my router), this setting is just called "Domain Name".

    If you then renew your DHCP lease on your Ubuntu box, you should see a search directive appear in /run/systemd/resolve/stub-resolv.conf:

    nameserver 127.0.0.53
    search your-domain
    

Now looking up web1 will expand it to web1.your-domain, which will then resolve using DNS.

$ nslookup web1
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   web1.your-domain
Address: 192.168.1.107

Note that if you use dig instead of nslookup, dig doesn't use the search path by default — use its +search option to enable that.

5
  • Before rebooting it looked up web1.mydomain.com just fine. But of course looking up just web1 didn't work. So i rebooted and for the life of me I have no idea where it is picking up the Comcast domain but now if I lookup web1 it responds with the correct IP but shows the comcast domain instead of my domain. It resolves so I am not too worried but what the heck????
    – schworak
    Sep 5 '18 at 11:52
  • @schworak Weird! Is your DHCP server also your Comcast modem? Do you see that domain showing up in /etc/resolv.conf or in the output of either nmcli -g all or systemd-resolve --status? Maybe try looking at what's in your DHCP lease? Sep 5 '18 at 16:31
  • It is not the comcast modem. I have a SysLink router with DDWRT running. The comcast settings are totally replaced. The comcast name shows up in the resolv.conf file which is auto-generated at boot up. I am not too worried about it but it is weird.
    – schworak
    Sep 6 '18 at 1:27
  • @LaurenceGonsalves Thanks a ton for the github issue link. I had found workarounds to the problem but this actually helped me figure out the root issue. Mar 15 '19 at 22:07
  • This is the correct answer. In case you came here to look for a solution where you use OpenWrt to set some CNAME for another local host, what you need to do is to set both the CNAME and target to be fully qualified. That is, assuming the default OpenWrt domain to be lan, the CNAME should be mycname.lan and the target mytarget.lan. Once you do that, you can resolve the CNAME as both mycname and mycname.lan. Jun 27 '20 at 0:30
1

Sometimes removing the symlink at /etc/resolv.conf is the right thing to do.

I recently migrated our authoritative DNS server to a newly built host, which (unlike the previous one) had systemd installed.

The results were terrible.

Although the rest of the world would see the correct results, the systemd resolver would serve stale results results to local services, and that would mess up those other services, in particular the internal mail forwarder.

dig @127.0.0.53 would show it bumping the TTL back up from time to time, so its cache would never expire; I suspect it was effectively querying itself, since the NS points at this host.

Running systemd-resolve --flush-caches did nothing.

Editing /etc/systemd/resolved.conf and setting DNSStubListener=no eventually stopped the resolver running, though it seems to pop up for a few seconds every now and then.

At least now the traditional services would actually get the right answers, but the ones built to perform name lookups over DBUS are irredeemably broken.

Now I understand why so many people hated systemd; it interferes other subsystems that it has no reason to touch, and there's no effective way to stop it from doing so. Even NIS wasn't broken as badly as this; why did systemd need to change it and make it worse?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.