DNS Resolve is not working on 18.04 server

Question

I have done some fairly extensive searching and can't seem to find the needle in the haystack that fixes this issue.

I have a server running Ubuntu 18.04

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:    18.04
Codename:   bionic

I am running LXC/LXD on the server currently with only a single container which is actually a 16.04 image. DNS works fine from within the container. I believe this eliminates any potential network issue as the problem.

In the 18.04 install the following happens when using nslookup

nslookup google.com
;; connection timed out; no servers could be reached

However when including a dns server directly as such I get a lookup to work. Again seemingly ruling out firewall/network issues

nslookup google.com 1.1.1.1
Server:     1.1.1.1
Address:    1.1.1.1#53

Non-authoritative answer:
Name:   google.com
Address: 172.217.5.238
Name:   google.com
Address: 2607:f8b0:4006:802::200e

As part of following tips/tricks/guides some of the following things I tried are below, as well as various outputs that might be useful in helping nail this down.

I modified the following file to look as such. I only added the name servers. I did this following one of the fixes out there.

$ cat /etc/netplan/50-cloud-init.yaml
network:
version: 2
ethernets:
    ens3:
        dhcp4: true
        match:
            macaddress: <redacted for post>
        set-name: ens3
        nameservers:
            addresses: [8.8.4.4, 8.8.8.8, 1.1.1.1, 1.1.0.0]

This did seem to add the dns servers to the device

sudo systemd-resolve --status
Global
      DNS Domain: openstacklocal
      DNSSEC NTA: 10.in-addr.arpa
                  16.172.in-addr.arpa
                  168.192.in-addr.arpa
                  17.172.in-addr.arpa
                  18.172.in-addr.arpa
                  19.172.in-addr.arpa
                  20.172.in-addr.arpa
                  21.172.in-addr.arpa
                  22.172.in-addr.arpa
                  23.172.in-addr.arpa
                  24.172.in-addr.arpa
                  25.172.in-addr.arpa
                  26.172.in-addr.arpa
                  27.172.in-addr.arpa
                  28.172.in-addr.arpa
                  29.172.in-addr.arpa
                  30.172.in-addr.arpa
                  31.172.in-addr.arpa
                  corp
                  d.f.ip6.arpa
                  home
                  internal
                  intranet
                  lan
                  local
                  private
                  test

Link 5 (vethTR4JCU)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 3 (lxdbr0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 2 (ens3)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 8.8.4.4
                      8.8.8.8
                      1.1.1.1
                      1.1.0.0
                      <redacted for post>
         DNS Domain: openstacklocal

Even with the dns servers listed there, no lookups are possible using either dig, or nslookup.

I did install resolvconf as part of a guide, though I think that was unnecessary and just proved to make a bigger mess.

$ ls -al /etc/resolv.conf 
lrwxrwxrwx 1 root root 29 Jan 29 12:55 /etc/resolv.conf -> ../run/resolvconf/resolv.conf

cat /run/resolvconf/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53
search openstacklocal

This is as far as I seem to be able to get. If I add valid nameservers (8.8.8.8, 8.8.4.4, 1.1.1.1, etc) to /run/resolveconf/resolv.conf file:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53
nameserver 8.8.8.8   # manually added in for testing
search openstacklocal

I can get lookups to work as shown below. If course as is stated in the file, these changes are overwritten on reboot.

nslookup google.com
Server:     8.8.8.8
Address:    8.8.8.8#53

Non-authoritative answer:
Name:   google.com
Address: 172.217.15.78
Name:   google.com
Address: 2607:f8b0:4004:810::200e

EDIT: output of apply command

sudo netplan --debug apply
** (generate:15710): DEBUG: 14:11:34.829: Processing input file /etc/netplan/50-cloud-init.yaml..
** (generate:15710): DEBUG: 14:11:34.830: starting new processing pass
** (generate:15710): DEBUG: 14:11:34.878: ens3: setting default backend to 1
** (generate:15710): DEBUG: 14:11:34.879: Generating output files..
** (generate:15710): DEBUG: 14:11:34.879: NetworkManager: definition ens3 is not for us (backend 1)
DEBUG:netplan generated networkd configuration exists, restarting networkd
DEBUG:no netplan generated NM configuration exists
DEBUG:ens3 not found in {}
DEBUG:Merged config:
network:
  bonds: {}
  bridges: {}
  ethernets:
    ens3:
      dhcp4: true
      match:
        macaddress: <redacted for post>
      nameservers:
        addresses:
        - 8.8.4.4
        - 8.8.8.8
        - 1.1.1.1
        - 1.1.0.0
      set-name: ens3
  vlans: {}
  wifis: {}

DEBUG:Skipping non-physical interface: lo
DEBUG:device ens3 operstate is up, not changing
DEBUG:Skipping non-physical interface: lxdbr0
DEBUG:Skipping non-physical interface: vethTR4JCU
DEBUG:{}
DEBUG:netplan triggering .link rules for lo
DEBUG:netplan triggering .link rules for ens3
DEBUG:netplan triggering .link rules for lxdbr0
DEBUG:netplan triggering .link rules for vethTR4JCU

EDIT: Requsted

sudo iptables -L -n -v

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     tcp  --  ens3   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8443 /* allow connection to lxd */
 2336  152K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    1    60 ACCEPT     tcp  --  lxdbr0 *       10.100.106.40        0.0.0.0/0            tcp dpt:22
 1279 73342 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 8207 2604K ACCEPT     all  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            /* generated for LXD network lxdbr0 */
 9496 3318K ACCEPT     all  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            /* generated for LXD network lxdbr0 */

Chain OUTPUT (policy ACCEPT 70 packets, 8606 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            tcp spt:53 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            udp spt:53 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            udp spt:67 /* generated for LXD network lxdbr0 */

Anyone know the link/solution to this problem. I'm at a loss.

See my answer askubuntu.com/questions/1113360/… Try adding the libnss-resolve package. The other workaround to the bugs listed (do add yourself to the bug "does this affect me?" list) is to use the nameserver(s) directly, cutting systemd-resolv out of the loop by changing /etc/resolv.conf — ubfan1, Commented Jan 29, 2019 at 19:58
are you able to contact local DNS cache? output of iptables -L -n -v? — virullius, Commented Jan 29, 2019 at 20:26

virullius · Accepted Answer · 2019-01-29 20:34:57Z

10

TL;DR: allow port 53 tcp & udp to lo interface.

Even though the default policy on INPUT is ACCEPT, there is a final rule that drops anything not yet accepted. The only rules accepting traffic on port 53 are on the lxdbr0 interface. You could blanket allow everything on lo interface or just allow ports as needed.

To push a rule to allow everything on lo interface ahead of the other rules:

iptables -I INPUT 1 -i lo -j ACCEPT

answered Jan 29, 2019 at 20:34

virullius

6511 gold badge5 silver badges15 bronze badges

I can't believe I missed that rule. It's even in my notes! Thank you!
– Dave
Commented Jan 29, 2019 at 20:36
Unfortunately, I went down the same path Dave did, but I am not having a problem with my iptables... This is fantastically hard to debug now
– jgitter
Commented Dec 5, 2019 at 20:49
I stumbled with the same and solved it by strict syntax in the /etc/netplan/99-netcfg-vmware.yaml: I added spaces around the nameservers and before the square brackets like this: addresses: [ 10.103.4.50,10.103.4.51 ] instead of addresses: [10.103.4.50,10.103.4.51].
– Tallandtree
Commented Jan 6, 2020 at 7:21
Sorry, I do not understand the answer - or how this could have solved Daves problem: in Dave's INPUT chain, there is an ACCEPT anything rule - two lines above the mentioned DROP anything rule. This rule should also have accepted packets for "lo" port 53. Can anyone please explain? Thanx.
– mats1995
Commented Jan 8, 2022 at 16:28
It's been a few years and I don't remember, but I think there was something funny about the wildcard not matching the loopback interface.
– virullius
Commented Jan 9, 2022 at 19:51

Add a comment |

JB. · Accepted Answer · 2020-07-01 14:02:49Z

5

Frankly the only correct answer to this modern b***** was to :

apt remove ifupdown
apt install cloud-init
# comment out settings in /etc/network/interfaces
# complete settings in /etc/netplan/config.yaml

# Apply settings or reboot
netplan apply

The removal of ifupdown is needed to make the DNS resolver function properly.

answered Jul 1, 2020 at 14:02

JB.

78910 silver badges20 bronze badges

cloud-init was missing...but not missing somehow for me
– dardo
Commented Sep 29, 2020 at 2:08
1

Yep. Worked for me after about a year of suffering intermittently DNS during which nothing else has ever worked. See this AskUbuntu question which was never resolved. I didn't comment out any settings, just installed cloud-init and removed ifupdown then did netplan apply.
– LondonRob
Commented Oct 16, 2020 at 16:05

Add a comment |

Goner · Accepted Answer · 2019-07-12 03:03:09Z

1

While upgrading from 16.04 to 18.04 I had this same issue. Tried the above plus many other solutions that revolved around the resolv.conf.

My true culprit: POSTFIX. Somehow/someway postfix was interfering with my DNS configuration. After removing postfix with the --auto-remove, magically the DNS started to resolve again.

I can now ping and apt-get again. What a wonderful day :)

answered Jul 12, 2019 at 3:03

Goner

112 bronze badges

Add a comment |

zgoda · Accepted Answer · 2019-09-09 19:25:55Z

1

In my case upgrade from 16.04 to 18.04 mysteriously disabled systemd-resolved service. Fix was a matter of enabling it again.

answered Sep 9, 2019 at 19:25

zgoda

1413 bronze badges

Add a comment |

Carlos Dagorret · Accepted Answer · 2019-01-29 19:27:58Z

0

After you have changed to NetPlan configuration. Run

sudo netplan apply

To expand the changes. To evaluate the changes, execute:

sudo netplan --debug apply

Update with more information.

I have this file configuration: /etc/netplan/01-netcfg.yaml

The parameters that you put are correct. Why not try:

sudo mv /etc/netplan/50-cloud-init.yaml /etc/netplan/01-netcfg.yaml

PS: make a backup.

edited Jan 29, 2019 at 19:27

answered Jan 29, 2019 at 19:08

Carlos Dagorret

6444 silver badges18 bronze badges

Those were done when I made the changes.
– Dave
Commented Jan 29, 2019 at 19:10
Made no difference. I didn't think it would as the configuration file is actually being picked up and used as shown in the added domain server output. There seems to be a disconnect between the resolv 127.0.0.53 and the actual nameservers. I'm just not sure how that's supposed to work to even know how to troubleshoot further.
– Dave
Commented Jan 29, 2019 at 19:35

Add a comment |

Eliah Kagan · Accepted Answer · 2019-12-22 23:57:56Z

0

You can just try to apply the Netplan configuration again. After an update on one of my docker nodes I lost the DNS resolution and just used the command below to re-apply the network configuration on the server:

sudo netplan apply

Everything worked as expected after that.

edited Dec 22, 2019 at 23:57

Eliah Kagan

119k57 gold badges323 silver badges504 bronze badges

answered Dec 22, 2019 at 22:07

Nikolay Andonov

1

Add a comment |

SwaJime · Accepted Answer · 2020-04-13 11:15:08Z

0

Following up on Dave's post. I started having this issue after installing postfix and did not want to uninstall postfix.

I found that adding 8.8.8.8 to resolv.conf as mentioned was a temporary fix, but then i found that it wasn't even necessary.

The fix that worked for me, believe it or not, was simply to add a blank line before 127.0.0.53 in resolv.conf.

answered Apr 13, 2020 at 11:15

SwaJime

1

Add a comment |

Stack Exchange Network

DNS Resolve is not working on 18.04 server

7 Answers 7

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged
networking
server
18.04
dns
.

Linked

Hot Network Questions

DNS Resolve is not working on 18.04 server

7 Answers 7

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged networkingserver18.04dns.

Linked

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
networking
server
18.04
dns
.