3

I have done some fairly extensive searching and can't seem to find the needle in the haystack that fixes this issue.

I have a server running Ubuntu 18.04

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:    18.04
Codename:   bionic

I am running LXC/LXD on the server currently with only a single container which is actually a 16.04 image. DNS works fine from within the container. I believe this eliminates any potential network issue as the problem.

In the 18.04 install the following happens when using nslookup

nslookup google.com
;; connection timed out; no servers could be reached

However when including a dns server directly as such I get a lookup to work. Again seemingly ruling out firewall/network issues

nslookup google.com 1.1.1.1
Server:     1.1.1.1
Address:    1.1.1.1#53

Non-authoritative answer:
Name:   google.com
Address: 172.217.5.238
Name:   google.com
Address: 2607:f8b0:4006:802::200e

As part of following tips/tricks/guides some of the following things I tried are below, as well as various outputs that might be useful in helping nail this down.

I modified the following file to look as such. I only added the name servers. I did this following one of the fixes out there.

$ cat /etc/netplan/50-cloud-init.yaml
network:
version: 2
ethernets:
    ens3:
        dhcp4: true
        match:
            macaddress: <redacted for post>
        set-name: ens3
        nameservers:
            addresses: [8.8.4.4, 8.8.8.8, 1.1.1.1, 1.1.0.0]

This did seem to add the dns servers to the device

sudo systemd-resolve --status
Global
      DNS Domain: openstacklocal
      DNSSEC NTA: 10.in-addr.arpa
                  16.172.in-addr.arpa
                  168.192.in-addr.arpa
                  17.172.in-addr.arpa
                  18.172.in-addr.arpa
                  19.172.in-addr.arpa
                  20.172.in-addr.arpa
                  21.172.in-addr.arpa
                  22.172.in-addr.arpa
                  23.172.in-addr.arpa
                  24.172.in-addr.arpa
                  25.172.in-addr.arpa
                  26.172.in-addr.arpa
                  27.172.in-addr.arpa
                  28.172.in-addr.arpa
                  29.172.in-addr.arpa
                  30.172.in-addr.arpa
                  31.172.in-addr.arpa
                  corp
                  d.f.ip6.arpa
                  home
                  internal
                  intranet
                  lan
                  local
                  private
                  test

Link 5 (vethTR4JCU)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 3 (lxdbr0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 2 (ens3)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 8.8.4.4
                      8.8.8.8
                      1.1.1.1
                      1.1.0.0
                      <redacted for post>
         DNS Domain: openstacklocal

Even with the dns servers listed there, no lookups are possible using either dig, or nslookup.

I did install resolvconf as part of a guide, though I think that was unnecessary and just proved to make a bigger mess.

$ ls -al /etc/resolv.conf 
lrwxrwxrwx 1 root root 29 Jan 29 12:55 /etc/resolv.conf -> ../run/resolvconf/resolv.conf

cat /run/resolvconf/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53
search openstacklocal

This is as far as I seem to be able to get. If I add valid nameservers (8.8.8.8, 8.8.4.4, 1.1.1.1, etc) to /run/resolveconf/resolv.conf file:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53
nameserver 8.8.8.8   # manually added in for testing
search openstacklocal

I can get lookups to work as shown below. If course as is stated in the file, these changes are overwritten on reboot.

nslookup google.com
Server:     8.8.8.8
Address:    8.8.8.8#53

Non-authoritative answer:
Name:   google.com
Address: 172.217.15.78
Name:   google.com
Address: 2607:f8b0:4004:810::200e

EDIT: output of apply command

sudo netplan --debug apply
** (generate:15710): DEBUG: 14:11:34.829: Processing input file /etc/netplan/50-cloud-init.yaml..
** (generate:15710): DEBUG: 14:11:34.830: starting new processing pass
** (generate:15710): DEBUG: 14:11:34.878: ens3: setting default backend to 1
** (generate:15710): DEBUG: 14:11:34.879: Generating output files..
** (generate:15710): DEBUG: 14:11:34.879: NetworkManager: definition ens3 is not for us (backend 1)
DEBUG:netplan generated networkd configuration exists, restarting networkd
DEBUG:no netplan generated NM configuration exists
DEBUG:ens3 not found in {}
DEBUG:Merged config:
network:
  bonds: {}
  bridges: {}
  ethernets:
    ens3:
      dhcp4: true
      match:
        macaddress: <redacted for post>
      nameservers:
        addresses:
        - 8.8.4.4
        - 8.8.8.8
        - 1.1.1.1
        - 1.1.0.0
      set-name: ens3
  vlans: {}
  wifis: {}

DEBUG:Skipping non-physical interface: lo
DEBUG:device ens3 operstate is up, not changing
DEBUG:Skipping non-physical interface: lxdbr0
DEBUG:Skipping non-physical interface: vethTR4JCU
DEBUG:{}
DEBUG:netplan triggering .link rules for lo
DEBUG:netplan triggering .link rules for ens3
DEBUG:netplan triggering .link rules for lxdbr0
DEBUG:netplan triggering .link rules for vethTR4JCU

EDIT: Requsted

sudo iptables -L -n -v

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     tcp  --  ens3   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8443 /* allow connection to lxd */
 2336  152K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    1    60 ACCEPT     tcp  --  lxdbr0 *       10.100.106.40        0.0.0.0/0            tcp dpt:22
 1279 73342 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 8207 2604K ACCEPT     all  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            /* generated for LXD network lxdbr0 */
 9496 3318K ACCEPT     all  --  lxdbr0 *       0.0.0.0/0            0.0.0.0/0            /* generated for LXD network lxdbr0 */

Chain OUTPUT (policy ACCEPT 70 packets, 8606 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            tcp spt:53 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            udp spt:53 /* generated for LXD network lxdbr0 */
    0     0 ACCEPT     udp  --  *      lxdbr0  0.0.0.0/0            0.0.0.0/0            udp spt:67 /* generated for LXD network lxdbr0 */

Anyone know the link/solution to this problem. I'm at a loss.

  • See my answer askubuntu.com/questions/1113360/… Try adding the libnss-resolve package. The other workaround to the bugs listed (do add yourself to the bug "does this affect me?" list) is to use the nameserver(s) directly, cutting systemd-resolv out of the loop by changing /etc/resolv.conf – ubfan1 Jan 29 at 19:58
  • 1
    are you able to contact local DNS cache? output of iptables -L -n -v? – mjb2kmn Jan 29 at 20:26
  • Updated question with requested output – Dave Jan 29 at 20:30
2

TL;DR: allow port 53 tcp & udp to lo interface.

Even though the default policy on INPUT is ACCEPT, there is a final rule that drops anything not yet accepted. The only rules accepting traffic on port 53 are on the lxdbr0 interface. You could blanket allow everything on lo interface or just allow ports as needed.

To push a rule to allow everything on lo interface ahead of the other rules:

iptables -I INPUT 1 -i lo -j ACCEPT
  • I can't believe I missed that rule. It's even in my notes! Thank you! – Dave Jan 29 at 20:36
0

After you have changed to NetPlan configuration. Run

sudo netplan apply

To expand the changes. To evaluate the changes, execute:

sudo netplan --debug apply

Update with more information.

I have this file configuration: /etc/netplan/01-netcfg.yaml

The parameters that you put are correct. Why not try:

sudo mv /etc/netplan/50-cloud-init.yaml /etc/netplan/01-netcfg.yaml

PS: make a backup.

  • Those were done when I made the changes. – Dave Jan 29 at 19:10
  • Made no difference. I didn't think it would as the configuration file is actually being picked up and used as shown in the added domain server output. There seems to be a disconnect between the resolv 127.0.0.53 and the actual nameservers. I'm just not sure how that's supposed to work to even know how to troubleshoot further. – Dave Jan 29 at 19:35
0

While upgrading from 16.04 to 18.04 I had this same issue. Tried the above plus many other solutions that revolved around the resolv.conf.

My true culprit: POSTFIX. Somehow/someway postfix was interfering with my DNS configuration. After removing postfix with the --auto-remove, magically the DNS started to resolve again.

I can now ping and apt-get again. What a wonderful day :)

New contributor
Goner is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.