Docker daemon does not bind to TCP6

Question

I have a composition with the default networking configuration and a definition such as:

services:

  myservice:
    image: someimage
    restart: always
    ports:
      - 80:80
      - 443:443

The service properly binds to IPv4 and can be accessed as expected. Since a week or two, the service does no longer bind to IPv6 which previously worked without problems. Running netstat -plnt shows me that the docker-proxy is not listening on IPv6 ports:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:6379            0.0.0.0:*               LISTEN      1936/docker-proxy
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1898/docker-proxy
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      811/systemd-resolve
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      1883/docker-proxy
tcp6       0      0 :::2377                 :::*                    LISTEN      1283/dockerd
tcp6       0      0 :::7946                 :::*                    LISTEN      1283/dockerd

So you can see that ports 80 and 443 are exposed on tcp but not tcp6. Searching for this issue I can only find the reversed problem (docker is listening on IPv6 but not IPv4).

When trying to bin the port with socat the port is reported in use (while netstat states it is not). Binding port 81 to the IPv4 address on port 80 allows me to access the server via IPv6 so there is no routing issue anywhere else.

sudo socat TCP6-LISTEN:80,fork TCP4:127.0.0.1:80
2021/01/13 16:08:50 socat[26572] E bind(5, {AF=10 [0000:0000:0000:0000:0000:0000:0000:0000]:80}, 28): Address already in use

docker inspect shows the following information:

"NetworkSettings": {
    "Bridge": "",
    "SandboxID": "d5fdebb4de954a4d7c1800490e44d0f53c4ee827775edb8ba286583e888eaa07",
    "HairpinMode": false,
    "LinkLocalIPv6Address": "",
    "LinkLocalIPv6PrefixLen": 0,
    "Ports": {
        "443/tcp": [
            {
                "HostIp": "0.0.0.0",
                "HostPort": "443"
            }
        ],
        "80/tcp": [
            {
                "HostIp": "0.0.0.0",
                "HostPort": "80"
            }
        ]
    },
    "SandboxKey": "/var/run/docker/netns/d5fdebb4de95",
    "SecondaryIPAddresses": null,
    "SecondaryIPv6Addresses": null,
    "EndpointID": "",
    "Gateway": "",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "IPAddress": "",
    "IPPrefixLen": 0,
    "IPv6Gateway": "",
    "MacAddress": "",
    "Networks": {
        "docker_default": {
            "IPAMConfig": null,
            "Links": null,
            "Aliases": [
                "d8acfbf724cf"
            ],
            "NetworkID": "87b6b52c779252614553040f217f9f2310ee3cce5f1a450f6a8210e8ea411b5a",
            "EndpointID": "a6bdf4d85641a043c25812ac0759a7ad872a3ee15ff7ea0e3ddf6b2405967737",
            "Gateway": "172.20.0.1",
            "IPAddress": "172.20.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "MacAddress": "02:42:ac:14:00:02",
            "DriverOpts": null
        }
    }
}

Answer 1

It seems to be a semi-intentional¹ change in Docker 20.10.2, see the related discussion here: https://github.com/moby/libnetwork/issues/2607. Looks like a fix is underway.

In the meantime, downgrading to 20.10.1 works for me:

sudo apt install docker-ce=5:20.10.1~3-0~ubuntu-focal \
                 docker-ce-cli=5:20.10.1~3-0~ubuntu-focal
sudo apt-mark hold docker-ce docker-ce-cli

---

¹ Semi-intentional because, apparently, this feature was never intended to be used this way. I’m as surprised as you are…

Answer 2

Fixed this by moving the docker ports and creating a socat proxy with systemd as described in this GitHub discussion. Steps are as follows:

Move the ports of your container, e.g. 8080:80 and 8443:443

Install socat

apt-get install socat

Create two systemd services that will use socat to listen on both IPv4 and IPv6 and forward the traffic to the IPv4 endpoint of your service:

nano /etc/systemd/system/socat-tcp-80.service

[Unit] 
Description=Socat TCP:80 
After=network.target

[Service] 
Type=simple 
User=root 
ExecStart=/usr/bin/socat TCP6-LISTEN:80,,su=nobody,fork,reuseaddr TCP4:127.0.0.1:8080
Restart=on-failure

[Install] 
WantedBy=multi-user.target

Save the file and create another one for port 443.

Start the services and enable them at boot:

systemctl start socat-tcp-80
systemctl start socat-tcp-443
systemctl enable socat-tcp-80
systemctl enable socat-tcp-443

You can use journalctl -feu socat-tcp-80 to fetch the logs of your service.

I don't know why the behavior of docker changed recently, but this workaround allows us to support IPv6 nevertheless.