0

"Temporary failure in (DNS) Name Resolution"

Background:

I work in healthcare security with approximately ~125 servers across a dozen clients across the US The appliances are both physical machines and VMs, 1u rack servers and small form factor appliances. While we originally worked on Ubuntu 16.04, we migrated to Ubuntu 20.04 in the last year.

These machines are running a popular open source SIEM. They operate with a management interface on one network interface/adapter, and a port mirror on a second network interface/adapter

One of the biggest changes in that move was how network configuration changed, from using interfaces files under /etc/network to using YAML files with Netplan. As a smaller company, it's just me maintaining the servers. As aforementioned, they are located at sites throughout the US, so I do not have physical access to 80%+ of these servers due to their geographic location. I perform any sort of maintenance and updates via Microsoft Azure, or through DWS, a RDS client. Over the last four years, everything has gone relatively smoothly. Recently, however, I've started to see the same issue occur at multiple sites. There is no common underlying characteristics about the affected machines: some were built within the last 30 days; others have been in place and operating without issue for over a year or two without previous issue.

The Issue:

Recently, many machines are losing network connectivity. Some have been operating over a year without any issues. They are unable to ping external IP addresses and return the error of "Temporary failure in name resolution". I have scoured dozens of posts that discuss what seems to be a very common issues and attempted to implement the common solutions. None have worked. I have recorded these different solutions and their results, alond with other data I thought might be pertinent below, in hopes that perhaps someone can advise me on how to correct this error.

NET/NET:

ping 8.8.8.8 works without issue

ping www.google.com returns "temporary failure in name resolution"

I CAN ping gateway and local assets


Ifconfig

enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.171.18.6  netmask 255.255.255.0  broadcast 10.171.18.255
        inet6 fe80::201:2eff:fea3:a56e  prefixlen 64  scopeid 0x20<link>
        ether 00:01:2e:a3:a5:6e  txqueuelen 1000  (Ethernet)
        RX packets 1040415  bytes 77072784 (77.0 MB)
        RX errors 0  dropped 36599  overruns 0  frame 0
        TX packets 1929257  bytes 81061339 (81.0 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.171.255.1  netmask 255.255.255.0  broadcast 10.171.255.255
        inet6 fe80::201:2eff:fea3:a56f  prefixlen 64  scopeid 0x20<link>
        ether 00:01:2e:a3:a5:6f  txqueuelen 1000  (Ethernet)
        RX packets 4184943678  bytes 2571645450748 (2.5 TB)
        RX errors 0  dropped 17  overruns 0  frame 0
        TX packets 604  bytes 52418 (52.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
  • All adapters are up and running
  • All adapters reflect the correct IPv4 address & netmask, corresponding to the netplan YAML file (see below)
  • Everything looks normal here. Received/transmitted traffic on first (enp1s0) adapter is as expected, received traffic on enp2s0, the port mirror, is also as expected.

/etc/netplan

drwxr-xr-x   2 root root  4096 Jul  7 11:48 .
drwxr-xr-x 146 root root 12288 Jun 29 13:44 ..
-rw-r--r--   1 root root   330 May 19 17:47 01-static-ip.yaml
  • Only one file is present, the configuration file 01-static-ip.yaml
  • This is currently chmoded 644
network:
   version: 2
   renderer: NetworkManager
   ethernets:
      enp1s0:
         dhcp4: no
         addresses:
            - 10.171.18.6/24
         nameservers:
            addresses: [10.30.3.29, 10.30.3.30]
         gateway4: 10.171.0.1
      enp2s0:
         dhcp4: no
         addresses:
            - 10.171.255.1/24
  • This is the contents of 01-static-ip.yaml
  • The IP addresses for both network adapters show correctly in IFCONFIG (above), as does the subnet mask (/24 / 255.255.255.0) for the management adapter

/etc/network

drwxr-xr-x   6 root root  4096 Jul  7 11:48 .
drwxr-xr-x 146 root root 12288 Jun 29 13:44 ..
drwxr-xr-x   2 root root  4096 May 19 14:56 if-down.d
drwxr-xr-x   2 root root  4096 May 19 14:56 if-post-down.d
drwxr-xr-x   2 root root  4096 May 19 14:56 if-pre-up.d
drwxr-xr-x   2 root root  4096 May 19 14:56 if-up.d
-rwxr-xr-x   1 root root     0 Jul 13 09:34 interfaces
  • I have tried sudo touch interfaces to create an empty interfaces file, and I have tried it with no interfaces file whatsoever
  • I am aware that you can technically still use an interfaces file for network configuration file under Ubuntu 20.04 but that Netplan is the preferred/recommended method for network configuration as of Ubuntu 18.04. For that reason I have NOT tried creating an old school interfaces file here and deleting the Netplan YAML file

/etc/resolv.conf

nameserver 127.0.0.53
options edns0 trust-ad
  • I have tried adding nameserver 10.30.3.29 before nameserver 127.0.0.53
  • I have tried removing nameserver 127.0.0.53 and only having nameserver 10.30.3.29
  • Neither worked

resolvectl status

Link 2 (enp1s0)
      Current Scopes: DNS       
DefaultRoute setting: yes       
       LLMNR setting: yes       
MulticastDNS setting: no        
  DNSOverTLS setting: no        
      DNSSEC setting: no        
    DNSSEC supported: no        
  Current DNS Server: 10.30.3.29
         DNS Servers: 10.30.3.29
                      10.30.3.30
          DNS Domain: ~.    
  • It appears DNS servers are properly configured here...

/etc/systemd/resolved.conf

[Resolve]
#DNS=
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#DNSOverTLS=no
#Cache=no-negative
#DNSStubListener=yes
#ReadEtcHosts=yes         
  • The most common solution seems to be to uncomment the first line here and add your DNS nameserver - this did not resolve situation.

nmcli -f ipv4.addresses,ipv4.dns,ipv4.gateway,IP4.ADDRESS,IP4.DNS,IP4.GATEWAY con show netplan-enp1s0

[NMCLI con show SPECIFIC]
ipv4.addresses:                         10.171.18.6/24
ipv4.dns:                               10.30.3.29,10.30.3.30
ipv4.gateway:                           10.171.0.1
IP4.ADDRESS[1]:                         10.171.18.6/24
IP4.DNS[1]:                             10.30.3.29
IP4.DNS[2]:                             10.30.3.30
IP4.GATEWAY:                            10.171.0.1
  • Again, just confirming all network information is correct

/usr/lib/NetworkManager/conf.d/10-globally-managed-devices.conf

[keyfile]
unmanaged-devices=*,except:type:wifi,except:type:gsm,except:type:cdma,except:type:ethernet
  • I've had issues in the past where I needed to manually edit this fiel to include "except:type:ethernet" but this is already done

systemctl status NetworkManager

<warn>  [1657208914.3825] ifupdown: interfaces file /etc/network/interfaces doesn't exist
  • This was generated when there was no interfaces file in /etc/network
  • As previously mentioned, I have tried this with and without this file present
  • upon creating an empty interfaces file and reloading NetworkManager, this problem disappeared
<warn>  [1657208914.4014] Error: failed to open /run/network/ifstate
  • I've seen this error previously, to. It usually is resolved by editing 10-globally-managed-devices.conf
<warn>  [1657208914.3941] device (enp1s0): connectivity: "/proc/sys/net/ipv4/conf/enp1s0/rp_filter" is set to "1". This might break connectivity checking for IPv4 on this device
  • I've also seen this error previously and tried editing this value to 0. Previously that had done the trick; here, there were no results.

Thank you in advance for your time and thoughts. I am happy to supply any additional information you'd like to see!

1
  • Look in dmesg output for any DNS fallback to UDP message. Look at the nsswitch package addition to change your /etc/nsswtich.conf hosts line -- might help. 22.04 changes things yet again...
    – ubfan1
    Commented Jul 16, 2022 at 16:39

1 Answer 1

0

A difference between Ubuntu 16.04 and Ubuntu 20.04 is the use of a local resolver by default, systemd-resolved. However, in servicing you note that you have tried disabling this resolver in your resolve.conf config, leaving only the 10.x.x.x server. This suggests that the problem is with the connection to your DNS server and not a problem of systemd-resolved itself. If possible, you should test with an Ubuntu 16.04 system, to rule out whether this is actually related to the client OS upgrade, or if something else happened to change at the same time introducing these transient failures.

The tcpdump tool may be useful for capturing network traces of the DNS traffic, to see what is happening at the protocol level: tcpdump -i enp1s0 -n -s 1500 port 53. If you use the -w option to write this output to a pcap file, you can further analyze with a tool such as ethereal.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .