link local (fe80::) addresses without scope make their way into /etc/resolv.conf and cause problems

Question

I'm having intermittent connectivity issues. An ipv6 link local address gets automatically added to my /etc/resolv.conf, and that seems to cause libc's resolver to fail resolving. I would like to know how to either prevent that address to be inserted there or find a suitable workaround.

My setup: I have a Ubuntu 14.04 desktop deployment setup with ipv4 and ipv6. It has only one wired connection (no wifi) to a lan port of a home router running OpenWrt. The desktop's networking is taken in charge by NetworkManager, which runs its own local copy of dnsmasq. All the network manager files in /etc are "stock", I haven't touched them.

When I reset my networking via network manager, everything works fine (but only for a few minutes). My working configuration of resolv.conf looks like so:

user@foo:/$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by     
resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
search lan

My router (192.168.1.1 or fe80::beef) also runs a copy of dnsmasq, and is configured to advertise 192.168.1.1 (itself) to its dhcp clients on v4. On v6, it periodically sends router advertisement messages (icmpv6.type == 134) with a recursive DNS Server option for fe80::beef. The router's dnsmasq DNS service is listening on both addresses: .1.1 and ::beef (the router's lan bridge ipv6 link address).

# working dns server. ran from the desktop.
user@foo:/$ dig google.com +time=1 @fe80::beef > /dev/null ; echo $?
0

At any time, if I go to "Connection Information" in NetworkManager, my primary DNS and router in Ipv4 is set to 182.168.1.1. The NetworkManager GUI shows no information under the "ipv6" section header -- but my nic receives ipv6 addresses (slaac and stateful dhcpv6 ones) which i can view with ip addr show.

The problem: After resetting my network via network manager (toggle "Enable Networking") and waiting (i.e. waiting until the next router advertisement message, i suspect), a new entry makes its way to /etc/resolv.conf:

user@foo:~$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver fe80::beef
nameserver 127.0.1.1
search lan

The problem is that once this happens, some userland tools (including firefox and google-chrome) will fail to resolve (non-cached) domain names.

As far as I understand, working with link local addresses requires a link scope to be mentioned explicitly. The following trace shows how connect fails without the link scope (default scope_id of 0).

user@foo:~$ strace ping google.com
execve("/bin/ping", ["ping", "google.com"], [/* 73 vars */]) = 0
...

stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=220, ...}) = 0
socket(PF_INET6, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET6, sin6_port=htons(53), inet_pton(AF_INET6, "fe80::beef", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EINVAL (Invalid argument)
close(3)                                = 0
socket(PF_INET6, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET6, sin6_port=htons(53), inet_pton(AF_INET6, "fe80::beef", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EINVAL (Invalid argument)
close(3)                                = 0
write(2, "ping: unknown host google.com\n", 29ping: unknown host google.com) = 29
exit_group(2)                           = ?
+++ exited with 2 +++

DNS fails, but I have connectivity otherwise:

user@foo:~$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=60 time=7.77 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=60 time=7.81 ms
....

user@foo:~$ ping6 2607:f8b0:400a:808::200e  # google.com AAAA
PING 2607:f8b0:400a:808::200e(2607:f8b0:400a:808::200e) 56 data bytes
64 bytes from 2607:f8b0:400a:808::200e: icmp_seq=1 ttl=57 time=7.94 ms
64 bytes from 2607:f8b0:400a:808::200e: icmp_seq=2 ttl=57 time=7.86 ms
...

Adding the scope (%eth0) to the end of the address in resolv.conf fixes the issue:

nameserver fe80::beef%eth0
nameserver 127.0.1.1
search lan

But of course, this change gets wiped the next time around.

Is there any way to:

• Force ipv4 only to query DNS (I don't think I'll run an ipv6-only setup any time soon)
• Specify a default interface scope in the resolver (i.e. "%eth0")
• Change the router's dnsmasq (or radvd, or rdnss) to not advertise its ipv6 address for DNS (only its ipv4)

Edit: failed fix attempt

If I move the /etc/resolv.conf symlink to /etc/resolv.conf.old and write my own static /etc/resolv.conf containing only the local dnsmasq server ip (nameserver 127.0.1.1), I find out that the file is still being modified by something else which adds a "search" line.

user@foo:~$ cat /etc/resolv.conf  # my new file, not the symlink
# Edited by hand to avoid using the ipv6 link local scopeless address
# check resolv.conf.old to see normal file
#nameserver fe80::beef
nameserver 127.0.1.1
search lan

After a bit of time:

user@foo:~$ cat /etc/resolv.conf  # my new file prefixed by something
search lan.
# Edited by hand to avoid using the ipv6 link local scopeless address
# check resolv.conf.old to see normal file
#nameserver fe80::beef
nameserver 127.0.1.1
search lan

After a reboot:

user@foo:~$ cat /etc/resolv.conf  # eh. same line added again.
search lan.
search lan.
# Edited by hand to avoid using the ipv6 link local scopeless address
# check resolv.conf.old to see normal file
#nameserver fe80::beef
nameserver 127.0.1.1
search lan

So. unless I start playing with chattr +i and other tricks to prevent that script (or whatever it is) from touching /etc/resolv.conf, I feel like this semi-static option isn't really clean. Accountability when these files are changed or logging about it would be a plus. syslog has nothing.

Note: In an attempt to hide some private info, I've redacted address suffixes and hostnames above

Answer 1

I'm answering my own question, and providing an explanation and patch for reference.

The problem is in the dhcp6c scripts, provided with package wide-dhcpv6-client. dhcp6c runs as a daemon on my system (started by /etc/init.d/wide-dhcpv6-client) and sends out DHCPv6 requests every few minutes. I don't think wide-dhcpv6-client is installed by default in ubuntu desktop. Its configuration (/etc/wide-dhcpv6/dhcp6c.conf) is set to invoke script /etc/wide-dhcpv6/dhcp6c-script when an update needs to be made. This script is buggy:

1. It ignores scope in the case of link-local addresses

2. In cases where /etc/resolv.conf is not managed by resolvconf (i.e. not a symlink), it results in search X lines being appended without checking if the lines are already present (then the file is ever growing).

Assuming you have a single interface being monitored by dhcp6c**, patching the file /etc/wide-dhcpv6/dhcp6c-script with the patch below will fix the problem:

--- /etc/wide-dhcpv6/dhcp6c-script.orig 2016-09-04 17:12:35.405042056 -0700
+++ /etc/wide-dhcpv6/dhcp6c-script.new  2016-09-04 22:57:05.213169351 -0700
@@ -6,20 +6,48 @@

 [ -f /etc/default/wide-dhcpv6-client ] && . /etc/default/wide-dhcpv6-client

+# Guess interface to use as scope_id. Ideally, dhcp6c would pass the
+# name of the interface on which the DHCPv6 reply was received.
+scope=""
+for IFACE in $INTERFACES; do
+    scope="$IFACE"
+    break;
+done
+
+# exact line match. grep would interpret domain periods as special
+# chars.  The naive loop avoids relying on other non-coreutils
+# commands (sed, awk, etc.).
+has_line () {
+    local line
+    while read line; do
+    if [ "$line" = "$1" ]; then
+       return 0
+    fi
+    done
+    return 1
+}
+    
+
 if [ -n "$new_domain_name" -o -n "$new_domain_name_servers" ]; then
     old_resolv_conf=/etc/resolv.conf
     new_resolv_conf=/etc/resolv.conf.dhcp6c-new
     rm -f $new_resolv_conf
     if [ -n "$new_domain_name" ]; then
-        echo search $new_domain_name >> $new_resolv_conf
+   has_line "search $new_domain_name" < $old_resolv_conf || {
+       echo "search $new_domain_name" >> $new_resolv_conf
+   }
     fi
     if [ -n "$new_domain_name_servers" ]; then
         for nameserver in $new_domain_name_servers; do
+       if [ -n "$scope" -a "${nameserver##fe80::}" != "$nameserver" ]; then
+          nameserver="$nameserver%$scope"
+       fi
+
             # No need to add an already existing nameserver
-            res=$(grep "nameserver $nameserver" $old_resolv_conf)
-            if [ -z "$res" ]; then
-                echo nameserver $nameserver >> $new_resolv_conf
-            fi
+       has_line "nameserver $nameserver" < $old_resolv_conf || {
+                echo "nameserver $nameserver" >> $new_resolv_conf
+       }
         done
     fi

**The code above will pick the first interface from the list of interfaces in /etc/default/wide-dhcpv6-client, and suffix it to any nameserver starting with fe80::. This is similar to what is done in /etc/dhcp/dhclient-enter-hooks.d/resolvconf. It's not ideal, especially if you have multiple interfaces. dhcp6c does not pass the name of the interface in the environment, so we can only guess. (bug filed: https://bugs.launchpad.net/ubuntu/+source/wide-dhcpv6/+bug/1620221 )

If you have multiple nics and need this to be more robust, you can instead configure /etc/wide-dhcpv6/dhcp6c.conf to start a different copy of the script depending on the interface name, and hardcode the name of the interface in the script.

Before I'd figured out the problem, I'd discovered two workarounds, which don't resolve the issue, but avoid the symptoms.

Workaround 1: disable dnsmasq local resolver.

The entries in resolv.conf are populated correctly when NetworkManager's dnsmasq service is disabled. So I lose its perks. Not perfect.

Followed steps in How can I disable the DNS that Network Manager uses? . Also, see list of security and performance implications of disabling the local resolver.

After restarting NM, the link local address had the scope added, correctly:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 192.168.1.1
nameserver fe80::beef%eth0
search lan

I'm not exactly sure why this works, but I suspect it is because both NetworkManager and dhcp6c write to resolvconv, and the update script favors fe80::beef%eth0 (passed by NetworkManager) vs the same address without the scope (passed by dhcp6c).

Workaround 2: Configure OpenWrt to advertise an additional dhcpv6 server as part of RDNSS. This server will take precedence over the link local one in /etc/resolv.conf. I feel this is a slightly better solution than workaround #1, because it preserves the local resolver (at least for ipv4), but it is more complicated.

1. Setup OpenWrt with an IPv6 ULA-Prefix, say fd00:cafe::/48. (Menu -> Network -> Interfaces -> bottom of page). The router will get a local ipv6 of fd00::cafe::1 on the lan. (I tried this on OpenWRT 15.05.1 chaos calmer).

2. Instruct odhcpd (which provides the dhcpv6 service on that version of openwrt) to also advertise IP as an (additional) DNS server. odhcpd (version 2015-11-19-01d3f...) offers a config flag called dhcp.lan.dns (flag description). In luci UI, this is under Menu -> Network -> Interfaces -> LAN (click edit) -> DHCP Server section -> IPv6 Settings tab -> Announced DNS Servers. Add router's ULA address to the field. I also added the router's ipv4 subnet address to the list of dns servers (192.168.1.1), for the sake of symmetry.

With this in place, odhcpd will advertise both addresses as part of RA, but NM, to our advantage, only writes the ULA address to resolv.conf. The link-local address is never added to resolv.conf:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver fd00:cafe::1
nameserver 127.0.1.1
search lan

Looking at the wireshark traces, when configured this way, the RA message from openwrt will list both dns server addresses, but its DHCPv6 replies only have the fd00:cafe::1 one, so the bug in dhcp6c scripts is avoided.

If you have a unix host on the network not running a local resolver (e.g. if you applied workaround 1 and 2 on ubuntu), resolv.conf would have all of the IPs:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 192.168.1.1
nameserver fd00:cafe::1
nameserver fe80::beef%eth0
search lan

The default algorithm in libc's resolver is to try each entry from resolv.conf in turn, timing out on each. So having multiple aliases to the same DNS server could cause additional delays when/if problems arise. In my case, that most likely means my whole router's down, not just its dns server, so it's a non-issue. but your mileage may vary.

extra I used the following script to debug changes made to resolvconf. This allowed me to figure out which daemon was sending the incorrect changes:

#!/bin/bash

# logs all interactions to /sbin/resolvconf into syslog
# this script is named /sbin/resolvconf, temporarily.
# moved resolvconv binary to resolvconf.real, temporarily.

BIN=/sbin/resolvconf.real
LOGGER=/usr/bin/logger

ppid () { ps -p ${1:-$$} -o ppid= 2>/dev/null | sed 's/ //g'; }

PROC=$$

"$LOGGER" "resolvconf.wrapper args: $@"

for ((i=0; i<4; i++)); do
    PROC=$(ppid $PROC)
    if [[ "$PROC" == 0 || "$PROC" == "" ]]; then
    break;
    fi
    if [[ "$i" -eq 0 ]]; then
    "$LOGGER" "resolvconf.wrapper invoked by: pid=$PROC $(/bin/ps -p $PROC -o command=)"
    else
    "$LOGGER" "resolvconf.wrapper child of: pid=$PROC $(/bin/ps -p $PROC -o command=)"
    fi
done

# peek at stdin (which contains the config)
if [[ "$1" == "-a" ]] && [[ -n "$2" ]]; then
    tmp=$(/bin/mktemp)
    while read REST; do
    logger "resolvconf.wrapper feeding: $REST"
    echo $REST >> "$tmp"
    done
    exec < "$tmp"
    rm "$tmp"
fi

exec -a /sbin/resolvconf "$BIN" "$@"

With this in place, syslog contains blocks like so:

Sep  3 02:32:29 calm logger: resolvconf.wrapper invoked by: pid=12610 sh -c /sbin/resolvconf -a NetworkManager
Sep  3 02:32:29 calm logger: resolvconf.wrapper child of: pid=11910 NetworkManager
Sep  3 02:32:29 calm logger: resolvconf.wrapper child of: pid=1 /sbin/init
Sep  3 02:32:29 calm logger: resolvconf.wrapper feeding: # Generated by NetworkManager
Sep  3 02:32:29 calm logger: resolvconf.wrapper feeding: domain base
Sep  3 02:32:29 calm logger: resolvconf.wrapper feeding: search base base.
Sep  3 02:32:29 calm logger: resolvconf.wrapper feeding: nameserver 127.0.1.1
...
Sep  3 02:14:23 calm logger: resolvconf.wrapper gparent: pid=1553 /usr/sbin/dhcp6c -Pdefault eth0
Sep  3 02:14:23 calm logger: resolvconf.wrapper invoked by: pid=15926 /bin/sh /etc/wide-dhcpv6/dhcp6c-script
Sep  3 02:14:23 calm logger: resolvconf.wrapper args: -a eth0
Sep  3 02:14:23 calm logger: resolvconf.wrapper feeding: search base.
Sep  3 02:14:23 calm logger: resolvconf.wrapper feeding: nameserver fd00:cafe::1