0

Server has 4 NICs and one bridge (irelevant for discussion) configured by netplan. After running netplan apply --debug I get following merged config:

DEBUG:Merged config:
network:
  bonds: {}
  bridges:
    br0:
      dhcp4: true
      dhcp6: true
      interfaces:
      - enp4s0f3
  ethernets:
      enp4s0f0:
      dhcp4: true
      dhcp6: false
      link-local: []
    enp4s0f1:
      dhcp4: false
      dhcp6: true
    enp4s0f2:
      dhcp4: true
      dhcp6: true
    enp4s0f3:
      dhcp4: false
      dhcp6: false

I tried to run netcat and listen on different IPs and for sake of simplicity here is case when only two NICs have enabled IPv4: enp4s0f0 and enp4s0f2.
Routing table is following:

route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.31.100.1     0.0.0.0         UG    100    0        0 enp4s0f2
0.0.0.0         10.31.100.1     0.0.0.0         UG    100    0        0 enp4s0f0
0.0.0.0         10.31.100.1     0.0.0.0         UG    100    0        0 br0
10.31.100.0     0.0.0.0         255.255.254.0   U     0      0        0 enp4s0f2
10.31.100.0     0.0.0.0         255.255.254.0   U     0      0        0 enp4s0f0
10.31.100.0     0.0.0.0         255.255.254.0   U     0      0        0 br0
10.31.100.1     0.0.0.0         255.255.255.255 UH    100    0        0 enp4s0f2
10.31.100.1     0.0.0.0         255.255.255.255 UH    100    0        0 enp4s0f0
10.31.100.1     0.0.0.0         255.255.255.255 UH    100    0        0 br0
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

As you see there are generated 3 default routes. Topmost default route uses interface enp4s0f2 and this is only case where server responds to client. Here is partial list of IPs

2: enp4s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 10:60:4b:9a:bd:7c brd ff:ff:ff:ff:ff:ff
    inet 10.31.100.70/23 brd 10.31.101.255 scope global dynamic enp4s0f0
       valid_lft 28703sec preferred_lft 28703sec
    inet6 fe80::1260:4bff:fe9a:bd7c/64 scope link
       valid_lft forever preferred_lft forever
4: enp4s0f2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 10:60:4b:9a:bd:7e brd ff:ff:ff:ff:ff:ff
    inet 10.31.100.63/23 brd 10.31.101.255 scope global dynamic enp4s0f2
       valid_lft 28703sec preferred_lft 28703sec
    inet6 fe80::1260:4bff:fe9a:bd7e/64 scope link
       valid_lft forever preferred_lft forever

I did run two tests when netcat was listening on IP of enp4s0f0 and IP if enp4s0f2. I was running tcpdump in both cases.
Here is first case, using enp4s0f0 IP 10.31.100.70

nc -l -p 33333 -s 10.31.100.70

Connect by client running on different machine

nc 10.31.100.70 33333

tcpdump shows following:

sudo tcpdump -i any -nn -vv port 33333
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
11:20:54.744837 IP (tos 0x2,ECT(0), ttl 114, id 5558, offset 0, flags [DF], proto TCP (6), length 52)
    10.47.240.156.41078 > 10.31.100.70.33333: Flags [SEW], cksum 0x2a91 (correct), seq 1636906657, win 8192, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0
11:20:54.744837 IP (tos 0x2,ECT(0), ttl 114, id 5558, offset 0, flags [DF], proto TCP (6), length 52)
    10.47.240.156.41078 > 10.31.100.70.33333: Flags [SEW], cksum 0x2a91 (correct), seq 1636906657, win 8192, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0

This continues on since client side does not get reply to open socket.
I stopped netcat and started to listen on enp4s0f2 IP 10.31.100.63

nc -l -p 33333 -s 10.31.100.63

and run client

nc 10.31.100.63 33333

Client connected and tcpdump shows socket opening answer from server to client.

sudo tcpdump -i any -nn -vv port 33333
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
16:51:39.033834 IP (tos 0x2,ECT(0), ttl 114, id 23066, offset 0, flags [DF], proto TCP (6), length 52)
    10.47.240.156.41180 > 10.31.100.63.33333: Flags [SEW], cksum 0xce53 (correct), seq 2777304710, win 8192, options [mss 1380,nop,wscale 8,nop,nop,sackOK], length 0
16:51:39.033906 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)
    10.31.100.63.33333 > 10.47.240.156.41180: Flags [S.], cksum 0x6950 (incorrect -> 0x1c2a), seq 467254207, ack 2777304711, win 64240, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
16:51:39.068784 IP (tos 0x0, ttl 114, id 23067, offset 0, flags [DF], proto TCP (6), length 40)
    10.47.240.156.41180 > 10.31.100.63.33333: Flags [.], cksum 0x53ed (correct), seq 1, ack 1, win 1024, length 0
16:52:18.202401 IP (tos 0x0, ttl 114, id 23069, offset 0, flags [DF], proto TCP (6), length 40)
    10.47.240.156.41180 > 10.31.100.63.33333: Flags [R.], cksum 0x57e9 (correct), seq 1, ack 1, win 0, length 0

To emphasize response: 10.31.100.63.33333 > 10.47.240.156.41180:. To me, it looks like no issue with routing table. When any other IP is used than last in netplan config, no response is sent to socket opening request.
I do not have clue what is going on and how to configure netplan to be able to listen on all addresses.
Plan is to use dedicated NIC to Docker, another for bridge and one for management. All NICs are connected to same switch, same virtual network. I tried to configure them and enable one by one in netplan and I was able to connect from remote computer to any IP (but only when this NIC was only IP or latest in generated netplan config file).

  • 2
    Just to better understand your issue: Why are you assigning "static" IPs via DHCP on four interfaces? – StrabagISS_Krall Jan 28 at 8:13
  • My goal is to have 3 DHCP and one static. There is one DNS entry. Static IP will use IP specified in A-record. Other interfaces will be configured for KVM usage. I will create bridge on one of those interfaces. Actually, 2 interfaces would be enough for now. One with static IP and one with DHCP used by bridge. My KVM virtual machines are not accessible and now I am investigating why. As first step, I try to access host on two NICs. – zdenko.s Jan 28 at 12:25
  • Please show your routing table in the case where it doesn't work ('ip route'). You might also compare with the case where it does work, to see what is different. – slangasek Jan 29 at 4:53
  • Now I added virtual bridge and bound it to network interface. Definition of bridge is last in netplan. It does not meter since netplan --debug apply generates configuration and sorts according to its own rule. Bridges are first. As I mentioned, only last NIC is accessible.Since bridge is first, I cannot access server using bridge IP. To test further, I disabled dhcp on every interface except bridge. Bridge gets IP from DHCP and now I can access server using bridge's IP. – zdenko.s Jan 29 at 5:13
  • @slangasek ip route when only br0 has enabled dhcp4 in netplan config file 'ip route default via 10.31.100.1 dev br0 proto dhcp src 10.31.100.68 metric 100 10.31.100.0/23 dev br0 proto kernel scope link src 10.31.100.68 10.31.100.1 dev br0 proto dhcp scope link src 10.31.100.68 metric 100' – zdenko.s Jan 29 at 5:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.