systemd-resolved not resolving specific domains

Question

I have a very strange problem with Ubuntu 18.04.1 Server, where the default resolver, systemd-resolved, isn't resolving some specific domain names.

The one it reliably fails on is stephenreescarter.net:

valorin@wp:~$ dig stephenreescarter.net

; <<>> DiG 9.11.3-1ubuntu1.3-Ubuntu <<>> stephenreescarter.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7015
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;stephenreescarter.net.         IN      A

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sun Jan 27 20:01:05 UTC 2019
;; MSG SIZE  rcvd: 50

But the domain itself is fine and works everywhere else:

valorin@wp:~$ dig stephenreescarter.net @1.1.1.1

; <<>> DiG 9.11.3-1ubuntu1.3-Ubuntu <<>> stephenreescarter.net @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45539
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;stephenreescarter.net.         IN      A

;; ANSWER SECTION:
stephenreescarter.net.  228     IN      A       104.28.2.92
stephenreescarter.net.  228     IN      A       104.28.3.92

;; Query time: 1 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Sun Jan 27 20:00:52 UTC 2019
;; MSG SIZE  rcvd: 82

And other domains work fine, so it's not simply a case of the server not being able to resolve everything:

valorin@wp:~$ dig google.com

; <<>> DiG 9.11.3-1ubuntu1.3-Ubuntu <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24208
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             148     IN      A       74.125.24.100
google.com.             148     IN      A       74.125.24.101
google.com.             148     IN      A       74.125.24.102
google.com.             148     IN      A       74.125.24.113
google.com.             148     IN      A       74.125.24.138
google.com.             148     IN      A       74.125.24.139

;; Query time: 2 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sun Jan 27 20:00:57 UTC 2019
;; MSG SIZE  rcvd: 135

Rebooting the system sometimes solves the problem, likewise with sudo systemd-resolve --flush-caches. However these don't always work, or sometimes need to be attempted multiple times before they start working.

I can reproduce this problem on a newly created Ubuntu 18.04.1 DigitalOcean droplet in the SGP1 region.

In all other ways, systemd-resolve seems to work, so I have no clue what is going on.

Update - debugging info

valorin@wp:~$ ls -al /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Oct  3 16:43 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
valorin@wp:~$ cat /run/resolvconf/resolv.conf
cat: /run/resolvconf/resolv.conf: No such file or directory
1 valorin@wp:~$ cat /run/systemd/resolve/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 67.207.67.2
nameserver 67.207.67.3

Answer 1

I think that your /etc/resolv.conf symlink is wrong.

Currently you show...

~$ ls -al /etc/resolv.conf

lrwxrwxrwx 1 root root 39 Oct 3 16:43 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

I believe that it should point to resolv.conf, not stub-resolv.conf. To change it, we'll do this...

sudo rm -i /etc/resolv.conf # remove old symlink

sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf # recreate symlink

See if that helps in any way.

Answer 2

The default installation of 18.04 seems to be missing the libnss-resolve package, the installation of which fixes the /etc/nsswitch.conf file so the hosts line looks like

hosts:          files mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] dns myhostname  

If you scan your /var/log/syslog file, you will probably see lines like:

Jan 27 09:33:15 leno systemd-resolved[931]: Using degraded feature set (UDP) for DNS server 192.168.1.1.
Jan 27 10:06:12 leno systemd-resolved[931]: Grace period over, resuming full feature set (UDP+EDNS0) for DNS server 192.168.1.1.  

These indicate that sometimes you are running at a reduced function set over UDP, and large outputs can overflow the usual buffer. See launchpad bugs 1804487 and 1805027. Other workarounds like redirecting the /etc/resolv.conf link from /run/systemd/resolv/stub-resolv.conf to the .../resolv.conf file basically cut systemd out of the loop, providing a nameserver directly.

---

You tested resolving with 1.1.1.1, but not a 67... ip. Try:

dig stephenreescarter.net @67.207.67.2  

If that fails, the problem is not within systemd-resolvd which is using that namesserver. That nameserver does not work for me, but maybe it's not public.