6

I think this is one of the widely discussed problem in SO Network. Most of the solutions are either out-dated or stale suggestions with latest Ubuntu (21.04)/Systemd changes.

Problem: VPN Changes from Host Network are not honored by Docker containers.

Tried Solutions:

  1. Docker daemon supports custom dns resolution.
    {
        "dns": ["172.17.0.1", "8.8.8.8", "8.8.4.4"]
    }

This solution is not working with Systemd DNS Resolution.

resolvectl status

Link 7 (docker0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 92 (tun0)
    Current Scopes: DNS
         Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 1.1.1.1
       DNS Servers: 1.1.1.1
        DNS Domain: consul vpn.net
  1. Installed dnsmasq and bind it to docker0 interface in /etc/dnsmasq.conf
interface=docker0
listen-address=172.17.0.1

If I use bind-dynamic, it fails with bind-dynamic and bind-interfaces can't be used together. I had to dig up from where bind-interfaces is being set

Finally found here:

cat /etc/dnsmasq.d/libvirt-daemon
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: /etc/dnsmasq.d/libvirt-daemon
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ bind-dynamic
   2   │ except-interface=virbr0

Changed line 1 bind-interfaces to bind-dynamic and didn't add explicitly in dnsmasq.conf.

This seems to be working post restarting docker and dnsmasq services. However post rebooting the system, this doesn't work again. It only works if I start services in the following order docker first (so that 172.17.0.1 comes available) and dnsmasq following that or I have to restart dnsmasq service explicitly to bind to 172.17.0.1 properly.

I didn't disable Systemd DNS resolver to avoid any future upgrade/integration issues with Network Manager and any other components.

Please suggest the solution

  1. Avoid restarting dnsmasq service every time post reboot
  2. Is there elegant solution instead of messing around SystemD/dnsmasq/docker. Simply one DNS resolution across all services/interfaces?
4
  • Are you following up on this? Jun 7, 2021 at 4:58
  • @ArturMeinild Yes. I didn't get the concrete working answer to be honest.
    – RaceBase
    Jun 7, 2021 at 5:50
  • Could you elaborate why my answer does not work? Maybe I misunderstood your question. Jun 7, 2021 at 6:51
  • @ArturMeinild 1. Its not possible to add DNS Config for each development environment. 2. When Docker is pointed to Host resolution, it should resolve using Host DNS Resolver. 3. Without customizing or binding DNS resolution for docker0 interface Systemd-resolver can't resolve DNS names for docker0 interface as Docker has it's own DNS loop back
    – RaceBase
    Jun 7, 2021 at 10:55

3 Answers 3

1

If I understand your question correctly, you could provide host DNS info for each container you run instead. I don't see why this shouldn't work with resolved or dnsmasq DNS resolution (although I use Unbound).

You should add the following to each container you run to provide the necessary DNS info. I assume that your DNS resolver is running on the address 172.17.0.1 (correct me if I'm wrong).

Using Docker CLI: (just a random example config)

docker run -d \
  --name <container-name> \
  --dns 172.17.0.1 \
  --dns 172.17.0.2 \ # your secondary DNS server if you have one
  --dns 8.8.8.8 \
  --dns 8.8.4.4 \
  -p <port>:<port> \
  -v /path/to/data/<container>:/path/to/data \
  --restart always \
  <repo>/<container>:<branch>

Using Docker Compose: (same random config)

version: '3'

  services:

    <service-name>:
      image: <repo>/<container>:<branch>
      container_name: <container-name>
      dns:
        - 172.17.0.1
        - 172.17.0.2 # your secondary DNS server if you have one
        - 8.8.8.8
        - 8.8.4.4
      ports:
        - <port>:<port>
      volumes:
        - /path/to/data/<container>:/path/to/data
      restart: always

I have done this with a couple of containers, and they resolve DNS using my internal DNS (my addresses are 10.10.2.2 and 10.10.2.4 and use Unbound, but I don't see why it should be any different).

This approach has at least 2 advantages in my opinion:

  • You don't need to customize or configure your DNS resolver beyond what you would normally do - it "just" needs to listen on a host (which could be the local machine or any other DNS host - 172.17.0.1 in this case, 172.17.0.2 if you have secondary).
  • You also don't need to customize your container with its own DNS solution, since DNS info is provided as parameters on container initialization (you could even have different containers using different DNS solutions if this was required).

The 1 "disadvantage" I can think of:

  • You need to provide the DNS info for each container you start - but if you save your container configuration, this only needs to be done once for each container, and the DNS configuration should in most cases be similar.

I believe this answer fits your criteria of:

  • Avoid restarting dnsmasq service every time post reboot
  • Is there elegant solution instead of messing around SystemD/dnsmasq/docker. Simply one DNS resolution across all services/interfaces?
1
  • I believe this is already being done based on what's described in the question. The docker daemon config specified does the equivalent of automatically passing the --dns flags to every docker call using the default bridge network.
    – mtalexan
    Jan 11, 2023 at 15:53
0

I would try the solution to run dnsmasq in docker like listed here:

https://blog.csainty.com/2016/09/running-dnsmasq-in-docker.html

where dnsmasq will read all the .conf files added to /etc/dnsmasq (a container restart is required to load changes).

1
  • The problem being solved with dnsmasq is almost certainly that the recursive resolver (dnsmasq) needs to be on the host system itself where the recursive resolution will use whatever the host system's current DNS config. By running dnsmasq in another container, he's isolated the host system DNS resolution again and no longer solved the original problem he added dnsmasq for in the first place.
    – mtalexan
    Jan 11, 2023 at 15:55
0

You seem to almost have it. You need two things for your overall solution:

  1. DNS for each container pointed to your host
  2. A DNS recursive resolver running on your host

You have both setup already, but I suspect your resolver isn't setup properly to work. The docker daemon config is implicitly doing the equivalent of passing --dns 172.17.0.1 --dns 8.8.8.8 --dns 8.8.4.4 to every container using the default docker network, so every container should already be querying the recursive resolver first. You also have dnsmasq running on your host system to provide a recursive resolver.

Your issue seems to be in how dnsmasq is operating; it doesn't seem to be accepting DNS requests from the docker network. I suspect it has to do with the explicit docker interface binding conflicting with the dynamic binding of the libvirt-daemon. I'm guessing you might be able to just remove your docker-specific config and leave the libvirt-daemon config and it will start working after a dnsmasq restart. The bind-dynamic should bind to all network interfaces except the ones specifically excluded: (except-interface=virbr0), which would theoretically include the docker0 bridge network.

To answer your specific criteria:

  1. The dnsmasq service should be enabled to automatically start on boot by unmasking the systemd service for it, which would resolve your restart issue. With bind-dynamic, it should handle changes in network environment automatically while running and not need to be manually restarted.

  2. Unfortunately there isn't really a solution for getting docker's default DNS resolver to operate like it probably should. Docker copies the DNS configuration from your host to your containers by default, but then routes actual networking thru your host. So DNS lookups run independently of the host, but the IP paths operate using the host. This is guaranteed to cause problems in most use cases where split/tunnel networking is used (corporate environments, VPNs, etc), but does allow DNS separation isolation for other use cases. You pretty much have to run some recursive resolver on the host that is exposed on at least the docker bridge network if you want to solve this, which isn't something most Ubuntu systems have running/available by default.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .