1

Summary: In a bridged KVM/QEMU configuration, network packets destined to guest VM do not get there.

Configuration: The host is an up to date Ubuntu 20.04.2 LTS server; The guest is any of 3 VMs, a very old 16.04 Ubuntu server, an old Ubuntu 20.04 desktop, and a brand new Ubuntu 21.04 desktop. The first 2 VMs are being converted from non-bridged, NAT'd, and the 3rd was created specifying bridged networking. Ultimately the VMs will get their IP addresses from the main LAN, via DHCP, but for now and for better debugging information, they are using static IP addresses.

The host bridge definition, /etc/netplan/01-netcfg.yaml, (this is one of many many attempts):

# This file describes the network interfaces available on your system
# For more information, see netplan(5).
network:
  version: 2
  renderer: networkd
  ethernets:
    enp3s0:
      dhcp4: no
  bridges:
    br0:
#      interfaces: [ enp3s0 ]
      dhcp4: yes
#      dhcp6: no
#      link-local: [ ]
      interfaces:
        - enp3s0
#      parameters:
#        stp: true
#        forward-delay: 4

The virtual stuff, /etc/libvirt/qemu/networks/br0.xml:

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh net-edit br0
or other application using the libvirt API.
-->

<network>
  <name>br0</name>
  <uuid>40a8752c-d074-4802-bae8-b0aef95d9c99</uuid>
  <forward mode='bridge'/>
  <bridge name='br0'/>
</network>

Notes: many versions of bridge .xml files have been tried, including different names, and different references use differing techniques. The Ubuntu Serverguide references this which says the name and bridge name have to be the same, but other references do not have them the same. Once the bare file has been created with nano, these commands:

virsh net-define br0.xml
virsh net-autostart br0
virsh net-start br0

were used to add and configure it. The default, NAT'd way was unlinked, from the autostart directory, so as not to. Eventually, it was undefined. The result:

$ virsh net-list --all
 Name      State      Autostart   Persistent
----------------------------------------------
 br0       active     yes         yes

At this point, there are no iptables rules at all after re-boot. However the VMs do not have network access. Note that some references mention special iptables rules and special attributes for the br_netfilter module, all of which have been tried. This question is already long enough without going into all the detail of variations tried here.

Debug Details: No matter the configuration variant, the base issue is always the same, VM destined packets do not seem to arrive at the host, at least as seen by tcpdump. However, broadcast type packets do arrive and do get to the client VM.

This example will use 192.168.111.59 (MAC: 52:54:00:60:ea:0e), the 16.04 server VM, and 192.168.111.132, a raspberry-pi, on the LAN. The 20.04 host server is at 192.168.111.136. The netmask is 24 bits, 255.255.255.0. The gateway and DHCP server is a Debian server (on which bridged guest VMs work fine, by the way).

First tpcudmp as seen from the raspberry-pi during ping:

doug@rpi2:~ $ sudo tcpdump -n -tttt -i eth0 ether host 52:54:00:60:ea:0e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
2021-04-23 08:33:19.363553 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 08:33:19.487239 IP 192.168.111.132 > 192.168.111.59: ICMP echo request, id 27848, seq 14, length 64
2021-04-23 08:33:20.363542 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 08:33:20.527250 IP 192.168.111.132 > 192.168.111.59: ICMP echo request, id 27848, seq 15, length 64
2021-04-23 08:33:21.567215 IP 192.168.111.132 > 192.168.111.59: ICMP echo request, id 27848, seq 16, length 64
2021-04-23 08:33:22.607228 IP 192.168.111.132 > 192.168.111.59: ICMP echo request, id 27848, seq 17, length 64
2021-04-23 08:33:23.372351 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 08:33:23.647228 IP 192.168.111.132 > 192.168.111.59: ICMP echo request, id 27848, seq 18, length 64
2021-04-23 08:33:24.371431 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 46

Observe the VM is sending out packets just fine, as seen via all the ARP activity. However it never replies to anything. Now lets watch the same activity from the host, noting tcpdump output is the same for any of interfaces br0 or enp3s0 or vnet0.

$ sudo tcpdump -n -tttt -i br0 ether host 52:54:00:60:ea:0e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
2021-04-23 08:40:38.837608 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:39.837159 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:40.837122 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:43.842985 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:44.840895 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:45.840991 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:48.848508 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:49.848895 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:50.848871 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:51.514011 ARP, Reply 192.168.111.59 is-at 52:54:00:60:ea:0e, length 28
2021-04-23 08:40:52.928400 ARP, Reply 192.168.111.59 is-at 52:54:00:60:ea:0e, length 28
2021-04-23 08:40:53.853881 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-23 08:40:54.852472 ARP, Request who-has 192.168.111.1 tell 192.168.111.59, length 28

Observe every so often the VM does respond, but later on we'll see that it is to a broadcast packet. It also seems as though a problem is that 192.168.111.1 isn't responding. It is, and for whatever reason, that packets are not seen here at the tcpdump level. Also notice no ICMP packets from the raspberry-pi. Now, show the gateway responding (this is "br0" on a different computer. EDIT: replaced with a better capture example, hence the different timestamps):

$ sudo tcpdump -n -tttt -e -i br0 ether host 52:54:00:60:ea:0e
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on br0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
2021-04-23 22:25:17.434415 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 22:25:17.434432 xx:xx:xx:xx:xx:xx > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at xx:xx:xx:xx:xx:xx, length 28
2021-04-23 22:25:20.440843 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 22:25:20.440859 xx:xx:xx:xx:xx:xx > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at xx:xx:xx:xx:xx:xx, length 28
2021-04-23 22:25:21.438316 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 22:25:21.438332 xx:xx:xx:xx:xx:xx > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at xx:xx:xx:xx:xx:xx, length 28
2021-04-23 22:25:22.438266 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 22:25:22.438283 xx:xx:xx:xx:xx:xx > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at xx:xx:xx:xx:xx:xx, length 28
2021-04-23 22:25:25.446312 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 22:25:25.446329 xx:xx:xx:xx:xx:xx > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at xx:xx:xx:xx:xx:xx, length 28
2021-04-23 22:25:26.446195 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-23 22:25:26.446211 xx:xx:xx:xx:xx:xx > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at xx:xx:xx:xx:xx:xx, length 28

Observe the VM outgoing packets, and ARP does complete. I can't figure out how to copy and paste from the VM, which I communicate with via VNC, but it shows some completed, but stale ARP entries in response to the ip neigh command and tcpdump shows some ARP and broadcast packets from the LAN.

Other information (MACs not relevant to this question have been hidden):

$ brctl show br0
bridge name     bridge id               STP enabled     interfaces
br0             8000.3c7c3f0d9983       no              enp3s0
                                                        vnet0
$ brctl showmacs br0
port no mac addr                is local?       ageing timer
  1     xx:xx:xx:xx:xx:xx       no                 0.00
  1     3c:7c:3f:0d:99:83       yes                0.00
  1     3c:7c:3f:0d:99:83       yes                0.00
  2     52:54:00:60:ea:0e       no                 1.68
  1     xx:xx:xx:xx:xx:xx       no                 2.14
  1     xx:xx:xx:xx:xx:xx       no                36.84
  1     xx:xx:xx:xx:xx:xx       no                89.57
  1     xx:xx:xx:xx:xx:xx       no               226.51
  1     xx:xx:xx:xx:xx:xx       no                13.28
  1     xx:xx:xx:xx:xx:xx       no               165.68
  1     xx:xx:xx:xx:xx:xx       no               165.68
  1     xx:xx:xx:xx:xx:xx       no               265.02
  1     xx:xx:xx:xx:xx:xx       no                27.62
  2     fe:54:00:60:ea:0e       yes                0.00
  2     fe:54:00:60:ea:0e       yes                0.00

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether 3c:7c:3f:0d:99:83 brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3c:7c:3f:0d:99:83 brd ff:ff:ff:ff:ff:ff
    inet 192.168.111.136/24 brd 192.168.111.255 scope global dynamic br0
       valid_lft 51547sec preferred_lft 51547sec
7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:60:ea:0e brd ff:ff:ff:ff:ff:ff

EDIT: interestingly, all ARP packets from my D-Link AC2600 Wi-Fi Gigabit Router, configured as a switch, always appear on the host and get to the VM and are replied to:

$ sudo tcpdump -n -tttt -e -i br0 ether host aa:aa:aa:aa:aa:aa
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
2021-04-23 22:45:51.463524 aa:aa:aa:aa:aa:aa > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.59 (ff:ff:ff:ff:ff:ff) tell 192.168.111.58, length 46
2021-04-23 22:45:51.463631 52:54:00:60:ea:0e > aa:aa:aa:aa:aa:aa, ethertype ARP (0x0806), length 42: Reply 192.168.111.59 is-at 52:54:00:60:ea:0e, length 28
2021-04-23 22:46:51.466955 aa:aa:aa:aa:aa:aa > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.59 (ff:ff:ff:ff:ff:ff) tell 192.168.111.58, length 46
2021-04-23 22:46:51.467030 52:54:00:60:ea:0e > aa:aa:aa:aa:aa:aa, ethertype ARP (0x0806), length 42: Reply 192.168.111.59 is-at 52:54:00:60:ea:0e, length 28
2021-04-23 22:47:51.466889 aa:aa:aa:aa:aa:aa > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.59 (ff:ff:ff:ff:ff:ff) tell 192.168.111.58, length 46
2021-04-23 22:47:51.466965 52:54:00:60:ea:0e > aa:aa:aa:aa:aa:aa, ethertype ARP (0x0806), length 42: Reply 192.168.111.59 is-at 52:54:00:60:ea:0e, length 28
2021-04-23 22:48:51.479096 aa:aa:aa:aa:aa:aa > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.59 (ff:ff:ff:ff:ff:ff) tell 192.168.111.58, length 46
2021-04-23 22:48:51.479178 52:54:00:60:ea:0e > aa:aa:aa:aa:aa:aa, ethertype ARP (0x0806), length 42: Reply 192.168.111.59 is-at 52:54:00:60:ea:0e, length 28

EDIT 3 - New configuration test: In an attempt to reduce the number of variables the following was done:

  • An about to be retired Ubuntu 16.04 server was powered up, providing a new isolated LAN.
  • the host Ubuntu 20.04 server was connected directly to the LAN side NIC of the 16.04 server. No switches were involved at all, just one long Ethernet cable.
  • Things were tested, and seemed to work fine. Access to everything was via ssh from my main LAN out through my main static WAN IP and back in via my test WAN static IP to the old 16.04 server. Then a chained ssh session from there to the 20.04 host server.
  • The Ubuntu 16.04 VM client was started on the host.
  • "ping" was attempted from the old 16.04 gateway server to the client.
  • The results were the same as the original configuration.

tcpdump on gateway old 16.04 server:

doug@DOUG-64:~$ sudo tcpdump -n -tttt -e -i enp2s0 ether host 52:54:00:60:ea:0e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
2021-04-26 15:10:00.701941 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-26 15:10:00.701965 00:19:b9:0d:af:fa > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at 00:19:b9:0d:af:fa, length 28
2021-04-26 15:10:01.699156 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-26 15:10:01.699169 00:19:b9:0d:af:fa > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at 00:19:b9:0d:af:fa, length 28
2021-04-26 15:10:02.699141 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-26 15:10:02.699154 00:19:b9:0d:af:fa > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at 00:19:b9:0d:af:fa, length 28
2021-04-26 15:10:05.707404 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-26 15:10:05.707417 00:19:b9:0d:af:fa > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at 00:19:b9:0d:af:fa, length 28
2021-04-26 15:10:06.707097 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-26 15:10:06.707110 00:19:b9:0d:af:fa > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at 00:19:b9:0d:af:fa, length 28
2021-04-26 15:10:07.707094 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.111.1 tell 192.168.111.59, length 46
2021-04-26 15:10:07.707107 00:19:b9:0d:af:fa > 52:54:00:60:ea:0e, ethertype ARP (0x0806), length 42: Reply 192.168.111.1 is-at 00:19:b9:0d:af:fa, length 28

tcpdump on 20.04 host server:

doug@s19:~$ sudo tcpdump -n -tttt -e -i br0 ether host 52:54:00:60:ea:0e
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
2021-04-26 15:11:35.801771 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-26 15:11:36.800497 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-26 15:11:37.800491 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-26 15:11:40.807062 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-26 15:11:41.804469 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-26 15:11:42.804444 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-26 15:11:45.812553 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-26 15:11:46.812405 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28
2021-04-26 15:11:47.812398 52:54:00:60:ea:0e > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.111.1 tell 192.168.111.59, length 28

As a side note: From my chained ssh session to the 20.04 host server, I can chain again and ssh to the VM client just fine.

Conclusion: Something is wrong right at the the Link Layer on the Ubuntu 20.04 server such that the incoming packets are not even "seen" by tcpdump, nor get to the VM guest. Diagram.

EDIT 4: Comparing to information provided by Christian Ehrhardt a potential difference is on my system is the br0 MAC list might be incorrect, with the fist byte replaced. Note: non relevant MAC deleted, 3 VMs running:

doug@s19:~$ brctl showmacs br0
port no mac addr                is local?       ageing timer
  1     3c:7c:3f:0d:99:83       yes                0.00  <<< enp3s0, br0
  1     3c:7c:3f:0d:99:83       yes                0.00  <<< enp3s0, br0
  4     52:54:00:22:2f:dc       no                 5.15  <<< VM 3
  2     52:54:00:60:ea:0e       no                 3.29  <<< VM 1
  3     52:54:00:60:ea:3e       no                12.67  <<< VM 2
  4     fe:54:00:22:2f:dc       yes                0.00  <<< vnet2
  4     fe:54:00:22:2f:dc       yes                0.00  <<< vnet2
  2     fe:54:00:60:ea:0e       yes                0.00  <<< vnet0
  2     fe:54:00:60:ea:0e       yes                0.00  <<< vnet0
  3     fe:54:00:60:ea:3e       yes                0.00  <<< vnet1
  3     fe:54:00:60:ea:3e       yes                0.00  <<< vnet1

For whatever reason Christian does not show a vnet, or whatever it is called on his system, connection to the bridge (I do not know it it is relevant or not):

$ brctl showmacs br0
port no mac addr        is local?   ageing timer
  2 52:54:00:48:40:69   no         2.36   <- Guest
  1 52:54:00:95:e4:2a   no         0.00   <- outside system
  1 52:54:00:9b:9b:0e   yes        0.00   <- Host
  1 52:54:00:9b:9b:0e   yes        0.00   <- Host

EDIT 5: Similar data to EDIT 4, but from Debian Server with 2 VMs running, which is working properly:

doug@s15:~$ sudo brctl showmacs br0
port no mac addr                is local?       ageing timer
  1     52:54:00:22:2f:dc       no                17.85
  2     52:54:00:27:1b:5e       no                18.48  <<< VM 1
  3     52:54:00:27:1b:ae       no                 2.14  <<< VM 2
  1     f4:8c:eb:c8:08:a0       no                18.48
  2     fe:71:fa:75:16:93       yes                0.00  <<< tap0 (VM1)
  2     fe:71:fa:75:16:93       yes                0.00  <<< tap0
  3     fe:e1:c5:2a:c7:e3       yes                0.00  <<< tap1 (VM2)
  3     fe:e1:c5:2a:c7:e3       yes                0.00  <<< tap1

EDIT 6: information from networkctl, the thing to note is that the Debian server shows "Master: br0" whereas the Ubuntu server doesn't:

Ubuntu:

doug@s19:~$ networkctl
IDX LINK   TYPE     OPERATIONAL SETUP
  1 lo     loopback carrier     unmanaged
  2 enp3s0 ether    enslaved    configured
  3 br0    bridge   routable    configured
  4 vnet0  ether    carrier     unmanaged

4 links listed.
doug@s19:~$ man networkctl
doug@s19:~$ networkctl^Cnetworkctl
doug@s19:~$ networkctl status vnet0
● 4: vnet0
             Link File: /usr/lib/systemd/network/99-default.link
          Network File: n/a
                  Type: ether
                 State: carrier (unmanaged)
                Driver: tun
            HW Address: fe:54:00:60:ea:0e
                   MTU: 1500 (min: 68, max: 65521)
  Queue Length (Tx/Rx): 1/1
      Auto negotiation: no
                 Speed: 10Mbps
                Duplex: full
                  Port: tp

Apr 30 07:40:51 s19 systemd-networkd[530]: vnet0: Link UP
Apr 30 07:40:51 s19 systemd-networkd[530]: vnet0: Gained carrier

Debian:

doug@s15:~$ networkctl
IDX LINK   TYPE     OPERATIONAL SETUP
  1 lo     loopback carrier     unmanaged
  2 enp3s0 ether    enslaved    configured
  3 enp1s0 ether    routable    configured
  4 br0    bridge   routable    configured
 10 tap0   ether    carrier     unmanaged

5 links listed.
doug@s15:~$ networkctl status tap0
● 10: tap0
             Link File: /usr/lib/systemd/network/99-default.link
          Network File: n/a
                  Type: ether
                 State: carrier (unmanaged)
                Driver: tun
            HW Address: fe:8a:6a:ce:18:9c
                   MTU: 1500 (min: 68, max: 65521)
                 QDisc: pfifo_fast
                Master: br0   <<<<< Different than Ubuntu
  Queue Length (Tx/Rx): 1/1
      Auto negotiation: no
                 Speed: 10Mbps
                Duplex: full
                  Port: tp

Question: What is wrong and how can I get bridged VMs working properly?

14
  • I had something rather like this last year - turned out to be a slowly-dying router. Not saying you have a similar cause, just a reminder to be open-minded.
    – user535733
    Apr 23 at 16:58
  • @user535733 : Thank you for your comment. In this case the router is the Debian server at 192.168.111.1. I do observe 28 byte payload ARP packets from it verses 46 byte payload ARP packets from some other computers that do get to the VM. There are also 3 switches involved. I will try moving cables around and to different ports on the switches. Apr 23 at 18:14
  • @user535733 : I did find a loose fitting cable between two switches that was causing that link to sometimes drop from 1 Gigabit to 100 Megabits per second. However, nothing effecting my main issue. I even hooked up a really really long debugging Ethernet cable, and eliminated 2 switches in the chain of things. Apr 25 at 20:15
  • Hi @DougSmythies - somewhat independent to your issue I agree that the "name" of the bridge in the libvirt XML doesn't have to be the same as the real bridge name. That wrong statement is in the netplan examples linked from the server guide. FYI: I've pinged the owner of that page for a discussion and maybe a change unless we find a case where it actually is important. Apr 26 at 5:19
  • Hi @DougSmythies I've seen this a few times when user try to bridge to wireless, but that doesn't seem the case for you (enp3s0 sounds like a normal nic). Still the problem that wireless has in general can happen in certain switch configs which is that they do not allow multiple MACs to be registered at a port. So all that gets through is a) direct pkgs for the host and b) broadcast which a bit looks like your case. The config name is different per switch manufacturer so I can't help with that, but can you check the peer for what it considers the port matching your enp3s0 to be registered for? Apr 26 at 5:35
1

We have the same kvm vlan bridge problem on PowerEdge R740 and Broadcomm 10GB NIC. What we found are

  Broadcom Adv. Dual 10Gb Ethernet 21.65.33.33 & kernel 5.4 -> bad
  Broadcom Adv. Dual 10Gb Ethernet 21.65.33.33 & kernel 5.5 -> good
  Broadcom Adv. Dual 10Gb Ethernet 21.80.16.95 & kernel 5.5 -> bad
  Broadcom Adv. Dual 10Gb Ethernet 21.80.16.95 & kernel 5.12 -> bad

However, firmware roll back won't work, so be careful.

0

The problem was the kernel. The host 20.04 server is not using the HWE stack, and therefore the current kernel is 5.4.0.72.

However everything works fine with all kernels tested from mainline 5.5-rc1 and higher, including 5.12-rc6 (I didn't get around to installing 5.12 yet). Some older 5.4 series kernels were also tested, including mainlines 5.4.0 and 5.4.117 and they all failed.

It seems kernel bisection does not work backwards, i.e. when the good kernel is after the bad kernel. However, if one just reverses the definition of good and bad, then maybe. Reference.

EDIT: This 20.04 server needs the HWE stack, because the hardware is to new for the 5.4 series kernel. I was unable to bisect the kernel, as there were too many git bisect skips required as too many kernels didn't boot or didn't even compile. This whole thing has been a rather colossal waste of time. Eventually I will delete this question.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.