2

I have a very strange problem with Ubuntu 18.04.1 Server, where the default resolver, systemd-resolved, isn't resolving some specific domain names.

The one it reliably fails on is stephenreescarter.net:

valorin@wp:~$ dig stephenreescarter.net

; <<>> DiG 9.11.3-1ubuntu1.3-Ubuntu <<>> stephenreescarter.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7015
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;stephenreescarter.net.         IN      A

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sun Jan 27 20:01:05 UTC 2019
;; MSG SIZE  rcvd: 50

But the domain itself is fine and works everywhere else:

valorin@wp:~$ dig stephenreescarter.net @1.1.1.1

; <<>> DiG 9.11.3-1ubuntu1.3-Ubuntu <<>> stephenreescarter.net @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45539
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;stephenreescarter.net.         IN      A

;; ANSWER SECTION:
stephenreescarter.net.  228     IN      A       104.28.2.92
stephenreescarter.net.  228     IN      A       104.28.3.92

;; Query time: 1 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Sun Jan 27 20:00:52 UTC 2019
;; MSG SIZE  rcvd: 82

And other domains work fine, so it's not simply a case of the server not being able to resolve everything:

valorin@wp:~$ dig google.com

; <<>> DiG 9.11.3-1ubuntu1.3-Ubuntu <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24208
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             148     IN      A       74.125.24.100
google.com.             148     IN      A       74.125.24.101
google.com.             148     IN      A       74.125.24.102
google.com.             148     IN      A       74.125.24.113
google.com.             148     IN      A       74.125.24.138
google.com.             148     IN      A       74.125.24.139

;; Query time: 2 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sun Jan 27 20:00:57 UTC 2019
;; MSG SIZE  rcvd: 135

Rebooting the system sometimes solves the problem, likewise with sudo systemd-resolve --flush-caches. However these don't always work, or sometimes need to be attempted multiple times before they start working.

I can reproduce this problem on a newly created Ubuntu 18.04.1 DigitalOcean droplet in the SGP1 region.

In all other ways, systemd-resolve seems to work, so I have no clue what is going on.

Update - debugging info

valorin@wp:~$ ls -al /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Oct  3 16:43 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
valorin@wp:~$ cat /run/resolvconf/resolv.conf
cat: /run/resolvconf/resolv.conf: No such file or directory
1 valorin@wp:~$ cat /run/systemd/resolve/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 67.207.67.2
nameserver 67.207.67.3
8
  • Do you have dnsmasq running? Show me ls -al /etc/resolv.conf. Report back to @heynnema
    – heynnema
    Jan 27, 2019 at 21:09
  • Also show me cat /run/resolvconf/resolv.conf and cat /run/systemd/resolve/resolv.conf. Please put all results as an edit to your question, and then ping me.
    – heynnema
    Jan 27, 2019 at 21:37
  • @heynnema debugging info added to question
    – Stephen RC
    Jan 28, 2019 at 21:05
  • You didn't say if you're running dnsmasq. Take a look at my answer and see if it helps at all. Report back.
    – heynnema
    Jan 28, 2019 at 21:32
  • Those 67... nameservers don't seem to be online. What does systemd-resolve --status show for DNS Servers and DNS Domain (towards the end, may have to hit space twice to advance a page).
    – ubfan1
    Jan 28, 2019 at 22:10

2 Answers 2

4

I think that your /etc/resolv.conf symlink is wrong.

Currently you show...

~$ ls -al /etc/resolv.conf

lrwxrwxrwx 1 root root 39 Oct 3 16:43 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

I believe that it should point to resolv.conf, not stub-resolv.conf. To change it, we'll do this...

sudo rm -i /etc/resolv.conf # remove old symlink

sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf # recreate symlink

See if that helps in any way.

1
  • It seems to be working at the moment, but I'll try this as soon as it breaks again and will let you know.
    – Stephen RC
    Jan 29, 2019 at 20:20
1

The default installation of 18.04 seems to be missing the libnss-resolve package, the installation of which fixes the /etc/nsswitch.conf file so the hosts line looks like

hosts:          files mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] dns myhostname  

If you scan your /var/log/syslog file, you will probably see lines like:

Jan 27 09:33:15 leno systemd-resolved[931]: Using degraded feature set (UDP) for DNS server 192.168.1.1.
Jan 27 10:06:12 leno systemd-resolved[931]: Grace period over, resuming full feature set (UDP+EDNS0) for DNS server 192.168.1.1.  

These indicate that sometimes you are running at a reduced function set over UDP, and large outputs can overflow the usual buffer. See launchpad bugs 1804487 and 1805027. Other workarounds like redirecting the /etc/resolv.conf link from /run/systemd/resolv/stub-resolv.conf to the .../resolv.conf file basically cut systemd out of the loop, providing a nameserver directly.


You tested resolving with 1.1.1.1, but not a 67... ip. Try:

dig stephenreescarter.net @67.207.67.2  

If that fails, the problem is not within systemd-resolvd which is using that namesserver. That nameserver does not work for me, but maybe it's not public.

2
  • Ah awesome, good to know it's a known bug. Installing libnss-resolve didn't fix the issue though :-( I'll keep an eye on those bugs, but so far nothing seems to help.
    – Stephen RC
    Jan 28, 2019 at 21:04
  • This may have solved my problem - it didn't fix it initially, but I haven't seen the issue recur within the last 24 hours. Will give it a bit more time before accepting to make sure it's working.
    – Stephen RC
    Jan 30, 2019 at 23:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .