0

Problem

Certain actions cause the whole LAN network to fail in resolving domains, the devices can't access anything outside the LAN network.

Example

When trying top pull a docker image from a self hosted Gitlab instance, every device on the network loses internet connection. There are other actions which trigger this effect.

The pull will eventually succeed but at speeds of <100Kb/s. Aborting the pull will cause all devices to resume normal operation within minutes.

Note

I am still able to ssh into the device while all the devices are unable to access the internet. I can also access any LAN samba shares.

The problem seems to only occur when the destination address is the public facing address of the LAN network. Lets say DNS records for example.com point to 99.99.99.99, which in this case would be the public facing address. Browsing to example.com will be fine from within the LAN. Pulling a docker image from registry.example.com will break the network.

Data

System:

$ lsb_release -a
Distributor ID: Ubuntu
Description:    Ubuntu 19.04
Release:        19.04
Codename:       disco

Ethernet controller:

$ lspci |grep Ethernet
13:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)

Routes:

$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         _gateway        0.0.0.0         UG    0      0        0 enp19s0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-aa8d8a399367
172.19.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-b47396405d84
172.20.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-2bac20a2ccb1
172.21.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-c1cedda32d4f
172.22.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-db04364eb1ec
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 enp19s0

Network interfaces:

$ lshw -C network
  *-network DISABLED
       description: Wireless interface
       product: AR9285 Wireless Network Adapter (PCI-Express)
       vendor: Qualcomm Atheros
       physical id: 0
       bus info: pci@0000:02:00.0
       logical name: wlp2s0
       version: 01
       serial: 74:2f:68:82:81:f6
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress bus_master cap_list ethernet physical wireless
       configuration: broadcast=yes driver=ath9k driverversion=5.0.0-27-generic firmware=N/A latency=0 link=no multicast=yes wireless=IEEE 802.11
       resources: irq:16 memory:f7d00000-f7d0ffff
  *-network
       description: Ethernet interface
       product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
       vendor: Realtek Semiconductor Co., Ltd.
       physical id: 0
       bus info: pci@0000:13:00.0
       logical name: enp19s0
       version: 06
       serial: 38:60:77:be:92:40
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix vpd bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=r8169 duplex=full firmware=rtl_nic/rtl8168e-2.fw ip=192.168.0.92 latency=0 link=yes multicast=yes port=MII speed=1Gbit/s
       resources: irq:19 ioport:c000(size=256) memory:f1404000-f1404fff memory:f1400000-f1403fff

Followed by a bunch of docker virtual interafeces.

Contents of sole netplan configuration file:

$ cat /etc/netplan/50-cloud-init.yaml
# This file is generated from information provided by
# the datasource.  Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}

# https://netplan.io/reference
network:
    version: 2
    renderer: networkd
    ethernets:
        enp19s0:
            wakeonlan: true
            addresses:
            - 192.168.0.92/24
            gateway4: 192.168.0.1
            nameservers:
                addresses:
                - 1.1.1.1
                - 8.8.8.8
                - 1.0.0.1
                - 8.8.4.4
            dhcp4: false
            dhcp6: false
            accept-ra: false

Network

https://i.imgur.com/Hics5je.jpg

  • The modem/router combo issued by my ISP is in red and yellow (one physical device).
  • The purple device is a wireless access point.
  • The blue devices are network switches.

What I tried

  • Disconnect all but two devices and test.
    • Connect these two devices directly to router/modem combo to rule out switch failure.
  • Turn off IPv6 on both Ubuntu devices.
    • Pass ipv6.disable=1 kernel parameter.
  • Update network interface driver.
    • $ apt install r8168-dkms
  • Set the correct date/timezone.
  • Remove nameservers from netplan configuration.
  • Looked at Wireshark capture.
    • Not sure what exactly to look for.
0

Solution

I am in no way a network engineer so take this advice lightly.

I fixed my problem by:

  1. Creating AAAA records for my domains.
  2. Re-enabling IPv6 support on all the devices.
  3. Creating IPv6 firewall rules to allow some traffic to the intended host; in other words, create a whitelist.

Cause

I am still not certain what caused the sudden DNS failures with IPv6 disabled. My best guess is my ISP changed something on their side (up to and including the modem/router).

Side notes

  • Positive: On a side note, all my services are now fully IPv6 compliant.
  • Negative: This probably only hides the real problem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.