1

I am preparing migration of an UBUNTU 16.04.7 server with 5 NIC's to 20.04. My testing server has 4 NIC's: 2 onboard and 2 as Intel Pro NIC. Configured in 00-installer-config.yaml. The first NIC is configured as:

  version: 2
  renderer: networkd
  ethernets:
    enp65s0f0:
      addresses: [10.1.11.230/24]
      gateway4: 10.1.11.11
      nameservers:
        addresses: [10.1.11.11, 1.1.1.1]
    enp193s0f0:
      optional: true
    enp193s0f1:
      optional: true
    enp65s0f1:
      optional: true

This part works. The next 3 NIC's are going to be bond0, with vlans on top and bridges:

  bonds:
    bond0:
      interfaces: [enp193s0f0, enp193s0f1, enp65s0f1]
      parameters:
        mode: 802.3ad
        lacp-rate: fast
        mii-monitor-interval: 100
        min-links: 1
  vlans:
    vlan1:
      id: 1
      link: bond0
    vlan11:
      id: 11
      link: bond0
    vlan240:
      id: 240
      link: bond0
  bridges:
    br100:
      interfaces: [vlan1]
    br11:
      interfaces: [vlan11]
    br240:
      interfaces: [vlan240]

The bond0 does not work. I have searched and tried all kinds of variations with ordering of NIC's, dhcp4: false, dhcp6: false, optional:true only using the IntelPro NIC's for the bonding. I have reviewed answers on this and other forums, but no solution so far yet.

  • What's wrong with my configuration? Anything missing?
  • Is the architecture I am aiming at still supported? I want my KVM guests to use a bridge so they can communicate with each other without needing the physical switch and without needing VLAN to be configured in the guests. This worked perfectly in Ub16.04.

My switch is a HP V1910-24G and has 4 other working trunks (LACP/802.3ad), with the Ub16.04 (4 NIC's)server and 3 QNAP NAS's with each a double NIC.

See the following outputs on my Ubuntu 20.04 with the above netplan config:

networkctl list
IDX LINK       TYPE     OPERATIONAL SETUP     
  1 lo         loopback carrier     unmanaged 
  2 enp65s0f0  ether    routable    configured
  3 enp65s0f1  ether    enslaved    configured
  4 enp193s0f0 ether    enslaved    configured
  5 enp193s0f1 ether    enslaved    configured
  6 bond0      bond     degraded    configured
  7 br100      bridge   degraded    configured
  8 br11       bridge   degraded    configured
  9 br240      bridge   degraded    configured
 10 vlan.240   vlan     carrier     unmanaged 
 11 vlan.11    vlan     carrier     unmanaged 
 12 vlan.1     vlan     carrier     unmanaged 
 13 virbr0     bridge   no-carrier  unmanaged 
 14 virbr0-nic ether    off         unmanaged 
 17 vnet0      ether    degraded    unmanaged 

 ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp65s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether d0:50:99:d4:bc:e3 brd ff:ff:ff:ff:ff:ff
3: enp65s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 16:3d:9c:75:6b:c5 brd ff:ff:ff:ff:ff:ff
4: enp193s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 16:3d:9c:75:6b:c5 brd ff:ff:ff:ff:ff:ff
5: enp193s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond0 state UP mode DEFAULT group default qlen 1000
    link/ether 16:3d:9c:75:6b:c5 brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 16:3d:9c:75:6b:c5 brd ff:ff:ff:ff:ff:ff
7: br100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 16:3d:9c:75:6b:c5 brd ff:ff:ff:ff:ff:ff
8: br11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 16:3d:9c:75:6b:c5 brd ff:ff:ff:ff:ff:ff
9: br240: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 16:3d:9c:75:6b:c5 brd ff:ff:ff:ff:ff:ff
10: vlan.240@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br240 state UP mode DEFAULT group default qlen 1000
    link/ether 16:3d:9c:75:6b:c5 brd ff:ff:ff:ff:ff:ff
11: vlan.11@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br11 state UP mode DEFAULT group default qlen 1000
    link/ether 16:3d:9c:75:6b:c5 brd ff:ff:ff:ff:ff:ff
12: vlan.1@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br100 state UP mode DEFAULT group default qlen 1000
    link/ether 16:3d:9c:75:6b:c5 brd ff:ff:ff:ff:ff:ff
13: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:f4:49:5c brd ff:ff:ff:ff:ff:ff
14: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:f4:49:5c brd ff:ff:ff:ff:ff:ff
17: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br11 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether fe:54:00:d2:56:7d brd ff:ff:ff:ff:ff:ff

 netplan --debug apply
** (generate:11473): DEBUG: 13:54:31.627: Processing input file /etc/netplan/00-installer-config.yaml..
** (generate:11473): DEBUG: 13:54:31.628: starting new processing pass
** (generate:11473): DEBUG: 13:54:31.628: We have some netdefs, pass them through a final round of validation
** (generate:11473): DEBUG: 13:54:31.628: vlan240: setting default backend to 1
** (generate:11473): DEBUG: 13:54:31.628: Configuration is valid
** (generate:11473): DEBUG: 13:54:31.628: bond0: setting default backend to 1
** (generate:11473): DEBUG: 13:54:31.628: Configuration is valid
** (generate:11473): DEBUG: 13:54:31.628: br11: setting default backend to 1
** (generate:11473): DEBUG: 13:54:31.628: Configuration is valid
** (generate:11473): DEBUG: 13:54:31.628: enp65s0f1: setting default backend to 1
** (generate:11473): DEBUG: 13:54:31.628: Configuration is valid
** (generate:11473): DEBUG: 13:54:31.628: enp65s0f0: setting default backend to 1
** (generate:11473): DEBUG: 13:54:31.628: Configuration is valid
** (generate:11473): DEBUG: 13:54:31.628: vlan11: setting default backend to 1
** (generate:11473): DEBUG: 13:54:31.628: Configuration is valid
** (generate:11473): DEBUG: 13:54:31.628: enp193s0f1: setting default backend to 1
** (generate:11473): DEBUG: 13:54:31.628: Configuration is valid
** (generate:11473): DEBUG: 13:54:31.628: br100: setting default backend to 1
** (generate:11473): DEBUG: 13:54:31.628: Configuration is valid
** (generate:11473): DEBUG: 13:54:31.628: vlan1: setting default backend to 1
** (generate:11473): DEBUG: 13:54:31.628: Configuration is valid
** (generate:11473): DEBUG: 13:54:31.628: enp193s0f0: setting default backend to 1
** (generate:11473): DEBUG: 13:54:31.628: Configuration is valid
** (generate:11473): DEBUG: 13:54:31.628: br240: setting default backend to 1
** (generate:11473): DEBUG: 13:54:31.628: Configuration is valid
** (generate:11473): DEBUG: 13:54:31.628: Generating output files..
** (generate:11473): DEBUG: 13:54:31.629: openvswitch: definition enp65s0f0 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: NetworkManager: definition enp65s0f0 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: openvswitch: definition enp193s0f0 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: NetworkManager: definition enp193s0f0 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: openvswitch: definition enp193s0f1 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: NetworkManager: definition enp193s0f1 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: openvswitch: definition enp65s0f1 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: NetworkManager: definition enp65s0f1 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: openvswitch: definition bond0 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: NetworkManager: definition bond0 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: openvswitch: definition vlan1 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: NetworkManager: definition vlan1 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: openvswitch: definition vlan11 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: NetworkManager: definition vlan11 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: openvswitch: definition vlan240 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: NetworkManager: definition vlan240 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: openvswitch: definition br100 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: NetworkManager: definition br100 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: openvswitch: definition br11 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: NetworkManager: definition br11 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: openvswitch: definition br240 is not for us (backend 1)
** (generate:11473): DEBUG: 13:54:31.629: NetworkManager: definition br240 is not for us (backend 1)
(generate:11473): GLib-DEBUG: 13:54:31.629: posix_spawn avoided (fd close requested) 
(generate:11473): GLib-DEBUG: 13:54:31.630: posix_spawn avoided (fd close requested) 
DEBUG:netplan generated networkd configuration changed, restarting networkd
DEBUG:enp65s0f0 not found in {}
DEBUG:enp193s0f0 not found in {'enp65s0f0': {'addresses': ['10.1.11.230/24'], 'gateway4': '10.1.11.11', 'nameservers': {'addresses': ['10.1.11.11', '1.1.1.1']}}}
DEBUG:enp193s0f1 not found in {'enp65s0f0': {'addresses': ['10.1.11.230/24'], 'gateway4': '10.1.11.11', 'nameservers': {'addresses': ['10.1.11.11', '1.1.1.1']}}, 'enp193s0f0': {'optional': True}}
DEBUG:enp65s0f1 not found in {'enp65s0f0': {'addresses': ['10.1.11.230/24'], 'gateway4': '10.1.11.11', 'nameservers': {'addresses': ['10.1.11.11', '1.1.1.1']}}, 'enp193s0f0': {'optional': True}, 'enp193s0f1': {'optional': True}}
DEBUG:br100 not found in {}
DEBUG:br11 not found in {'br100': {'interfaces': ['vlan1']}}
DEBUG:br240 not found in {'br100': {'interfaces': ['vlan1']}, 'br11': {'interfaces': ['vlan11']}}
DEBUG:bond0 not found in {}
DEBUG:vlan1 not found in {}
DEBUG:vlan11 not found in {'vlan1': {'id': 1, 'link': 'bond0'}}
DEBUG:vlan240 not found in {'vlan1': {'id': 1, 'link': 'bond0'}, 'vlan11': {'id': 11, 'link': 'bond0'}}
DEBUG:Merged config:
network:
  bonds:
    bond0:
      interfaces:
      - enp193s0f0
      - enp193s0f1
      - enp65s0f1
      parameters:
        lacp-rate: fast
        mii-monitor-interval: 100
        min-links: 1
        mode: 802.3ad
  bridges:
    br100:
      interfaces:
      - vlan1
    br11:
      interfaces:
      - vlan11
    br240:
      interfaces:
      - vlan240
  ethernets:
    enp193s0f0:
      optional: true
    enp193s0f1:
      optional: true
    enp65s0f0:
      addresses:
      - 10.1.11.230/24
      gateway4: 10.1.11.11
      nameservers:
        addresses:
        - 10.1.11.11
        - 1.1.1.1
    enp65s0f1:
      optional: true
  ovs_ports: {}
  vlans:
    vlan1:
      id: 1
      link: bond0
    vlan11:
      id: 11
      link: bond0
    vlan240:
      id: 240
      link: bond0
  wifis: {}

DEBUG:no netplan generated NM configuration exists
DEBUG:enp65s0f0 not found in {}
DEBUG:enp193s0f0 not found in {'enp65s0f0': {'addresses': ['10.1.11.230/24'], 'gateway4': '10.1.11.11', 'nameservers': {'addresses': ['10.1.11.11', '1.1.1.1']}}}
DEBUG:enp193s0f1 not found in {'enp65s0f0': {'addresses': ['10.1.11.230/24'], 'gateway4': '10.1.11.11', 'nameservers': {'addresses': ['10.1.11.11', '1.1.1.1']}}, 'enp193s0f0': {'optional': True}}
DEBUG:enp65s0f1 not found in {'enp65s0f0': {'addresses': ['10.1.11.230/24'], 'gateway4': '10.1.11.11', 'nameservers': {'addresses': ['10.1.11.11', '1.1.1.1']}}, 'enp193s0f0': {'optional': True}, 'enp193s0f1': {'optional': True}}
DEBUG:br100 not found in {}
DEBUG:br11 not found in {'br100': {'interfaces': ['vlan1']}}
DEBUG:br240 not found in {'br100': {'interfaces': ['vlan1']}, 'br11': {'interfaces': ['vlan11']}}
DEBUG:bond0 not found in {}
DEBUG:vlan1 not found in {}
DEBUG:vlan11 not found in {'vlan1': {'id': 1, 'link': 'bond0'}}
DEBUG:vlan240 not found in {'vlan1': {'id': 1, 'link': 'bond0'}, 'vlan11': {'id': 11, 'link': 'bond0'}}
DEBUG:Merged config:
network:
  bonds:
    bond0:
      interfaces:
      - enp193s0f0
      - enp193s0f1
      - enp65s0f1
      parameters:
        lacp-rate: fast
        mii-monitor-interval: 100
        min-links: 1
        mode: 802.3ad
  bridges:
    br100:
      interfaces:
      - vlan1
    br11:
      interfaces:
      - vlan11
    br240:
      interfaces:
      - vlan240
  ethernets:
    enp193s0f0:
      optional: true
    enp193s0f1:
      optional: true
    enp65s0f0:
      addresses:
      - 10.1.11.230/24
      gateway4: 10.1.11.11
      nameservers:
        addresses:
        - 10.1.11.11
        - 1.1.1.1
    enp65s0f1:
      optional: true
  ovs_ports: {}
  vlans:
    vlan1:
      id: 1
      link: bond0
    vlan11:
      id: 11
      link: bond0
    vlan240:
      id: 240
      link: bond0
  wifis: {}

DEBUG:Skipping non-physical interface: lo
DEBUG:device enp65s0f0 operstate is up, not changing
DEBUG:Skipping composite member enp65s0f1
DEBUG:Skipping composite member enp193s0f0
DEBUG:Skipping composite member enp193s0f1
DEBUG:Skipping non-physical interface: bond0
DEBUG:Skipping non-physical interface: br100
DEBUG:Skipping non-physical interface: br11
DEBUG:Skipping non-physical interface: br240
DEBUG:Skipping non-physical interface: vlan.240
DEBUG:Skipping non-physical interface: vlan.11
DEBUG:Skipping non-physical interface: vlan.1
DEBUG:Skipping non-physical interface: virbr0
DEBUG:Skipping non-physical interface: virbr0-nic
DEBUG:Skipping non-physical interface: vnet0
DEBUG:{}
DEBUG:netplan triggering .link rules for lo
DEBUG:netplan triggering .link rules for enp65s0f0
DEBUG:netplan triggering .link rules for enp65s0f1
DEBUG:netplan triggering .link rules for enp193s0f0
DEBUG:netplan triggering .link rules for enp193s0f1
DEBUG:netplan triggering .link rules for bond0
DEBUG:netplan triggering .link rules for br100
DEBUG:netplan triggering .link rules for br11
DEBUG:netplan triggering .link rules for br240
DEBUG:netplan triggering .link rules for vlan.240
DEBUG:netplan triggering .link rules for vlan.11
DEBUG:netplan triggering .link rules for vlan.1
DEBUG:netplan triggering .link rules for virbr0
DEBUG:netplan triggering .link rules for virbr0-nic
DEBUG:netplan triggering .link rules for vnet0
DEBUG:enp65s0f0 not found in {}
DEBUG:enp193s0f0 not found in {'enp65s0f0': {'addresses': ['10.1.11.230/24'], 'gateway4': '10.1.11.11', 'nameservers': {'addresses': ['10.1.11.11', '1.1.1.1']}}}
DEBUG:enp193s0f1 not found in {'enp65s0f0': {'addresses': ['10.1.11.230/24'], 'gateway4': '10.1.11.11', 'nameservers': {'addresses': ['10.1.11.11', '1.1.1.1']}}, 'enp193s0f0': {'optional': True}}
DEBUG:enp65s0f1 not found in {'enp65s0f0': {'addresses': ['10.1.11.230/24'], 'gateway4': '10.1.11.11', 'nameservers': {'addresses': ['10.1.11.11', '1.1.1.1']}}, 'enp193s0f0': {'optional': True}, 'enp193s0f1': {'optional': True}}
DEBUG:br100 not found in {}
DEBUG:br11 not found in {'br100': {'interfaces': ['vlan1']}}
DEBUG:br240 not found in {'br100': {'interfaces': ['vlan1']}, 'br11': {'interfaces': ['vlan11']}}
DEBUG:bond0 not found in {}
DEBUG:vlan1 not found in {}
DEBUG:vlan11 not found in {'vlan1': {'id': 1, 'link': 'bond0'}}
DEBUG:vlan240 not found in {'vlan1': {'id': 1, 'link': 'bond0'}, 'vlan11': {'id': 11, 'link': 'bond0'}}
DEBUG:Merged config:
network:
  bonds:
    bond0:
      interfaces:
      - enp193s0f0
      - enp193s0f1
      - enp65s0f1
      parameters:
        lacp-rate: fast
        mii-monitor-interval: 100
        min-links: 1
        mode: 802.3ad
  bridges:
    br100:
      interfaces:
      - vlan1
    br11:
      interfaces:
      - vlan11
    br240:
      interfaces:
      - vlan240
  ethernets:
    enp193s0f0:
      optional: true
    enp193s0f1:
      optional: true
    enp65s0f0:
      addresses:
      - 10.1.11.230/24
      gateway4: 10.1.11.11
      nameservers:
        addresses:
        - 10.1.11.11
        - 1.1.1.1
    enp65s0f1:
      optional: true
  ovs_ports: {}
  vlans:
    vlan1:
      id: 1
      link: bond0
    vlan11:
      id: 11
      link: bond0
    vlan240:
      id: 240
      link: bond0
  wifis: {}


I am at the end of my options and ready to remove cloud-init/netplan and revert to the working configuration I had in 16.04. Thanks for reading this far.

Frank

2
  • Because this seems unsupported by netplan we follow this guide to remove netplan and revert to the ifupdown: linuxconfig.org/… – HeuvelTop Nov 10 '20 at 14:52
  • From the manual of the switch: Link aggregation (trunking): groups together up to 8 ports automatically using Link Aggregation Control Protocol (LACP), or manually, to form an ultra-high-bandwidth connection to the network backbone; helps prevent traffic bottlenecks. My new server exceeds that max, that I earlier interpreted as 8 ports per trunk, but it is 8 LACP ports in total. I will up date the above config with the working one. – HeuvelTop Nov 15 '20 at 14:23
0

I think your vlans are named incorrectly, they needs to contain interface name

  vlans:
    bond0.1:
      id: 1
      link: bond0
      ...

And I am assuming you removed addresses/dhcp on purpose from the example.

Update - sorry, I confused myself. You need to specify vlan number in bridge names, this is what is actually working for me (it is a virtual bridge for internal communication, no interfaces attached)

network:
  bridges:
    maasbr0:
      addresses: [ 10.244.40.1/21, ]
  vlans:
    maasbr0.2678:
      link: maasbr0
      id: 2678
      addresses: [ 10.244.8.1/24 ]
    maasbr0.2733:
      link: maasbr0
      id: 2733
      addresses: [ 192.168.33.1/25 ]
    maasbr0.2734:
      link: maasbr0
      id: 2734
      addresses: [ 10.244.32.1/21 ]
    maasbr0.2735:
      link: maasbr0
      id: 2735
      addresses: [ 192.168.35.1/26 ]
    maasbr0.2736:
      link: maasbr0
      id: 2736
      addresses: [ 192.168.36.1/26 ]
5
  • I have tested this. The vlans have changed from 'carrier' to 'enslaved' and from 'unmannaged' to 'configured'. Bond0 stays 'degraded'. Also the bridges are still 'degraded'. – HeuvelTop Nov 9 '20 at 20:37
  • Thank you for looking into to this, but this does not reflect the architecture I am aiming at. After looking very hard at the netplan documentation with a few people and searching the net we have decided to remove netplan. – HeuvelTop Nov 10 '20 at 14:51
  • It is certainly up to you, but I think that is not the right approach, netplan is here to stay. – marosg Nov 11 '20 at 15:03
  • I can return to it when the documentations are complete and useful. – HeuvelTop Nov 11 '20 at 15:59
  • To be clear about my setup: it works without the bond. That is a Bridge on Vlan on NIC. – HeuvelTop Nov 13 '20 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.