1

Server : Ubuntu Server 18.04.4

Router : pfSense 2.4.4

Switch : tp-link TL-SG1016DE

The Short : How do I configure multiple NIC on the same subnet/VLAN? Would like traffic that goes in one NIC to go out the same interface. This was working not sure what happened.

The Long:

Not sure what happened but my configuration seems to have borked itself and I am unable to ping or access the onboard NIC enp9s0. I'm at a loss as it was working before.

Accessing the VM's that are behind enp4s0f1 seem to be functioning correctly.

Attempts to PING or SSH into enp9s0 (192.168.2.2) fails.

If I delete the the ARP entry on pfSense for 192.168.2.2 then I am able to PING or SSH into enp9s0 (192.168.2.2) but will drop off shortly there after.

If however I then start a PING from within 192.168.2.2, after SSHing in, out to to another device on the LAN, the the connection remains stable.

However viewing the ARP table on pfSense now show the wrong MAC address associated with 192.168.2.2. It show the MAC address of enp4s0f1 rather than enp9s0.

Furthermore the ARP on the HOST shows matching hardware address that I have not seen before.

EDIT: The ARP on the HOST is showing the MAC address of the NIC on pfSense.

The three following NIC's are installed:

onboard : enp9s0 ( MACADDRESS:ad )

dual port : enp4s0f0 ( MACADDRESS:74 ) ; enp4s0f1 ( MACADDRESS:75 )

single port : enp7s0 ( MACADDRESS:ba )


#arp

Address                  HWtype  HWaddress           Flags Mask            Iface
HOSTNAME.localdomain              (incomplete)                              enp4s0f1
_gateway                 ether   MACADDRESS:e1   C                            enp4s0f1
_gateway                 ether   MACADDRESS:e1   C                            enp9s0


    enp4s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
                inet 192.168.2.4  netmask 255.255.255.0  broadcast 192.168.2.255
                ether MACADDRESS:74  txqueuelen 1000  (Ethernet)
                RX packets 46104  bytes 3302633 (3.3 MB)
                RX errors 0  dropped 30263  overruns 0  frame 0
                TX packets 821  bytes 66602 (66.6 KB)
                TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
                device memory 0xf6420000-f643ffff  

        enp4s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
                inet 192.168.2.5  netmask 255.255.255.0  broadcast 192.168.2.255
                ether MACADDRESS:75  txqueuelen 1000  (Ethernet)
                RX packets 106881  bytes 10461480 (10.4 MB)
                RX errors 0  dropped 30268  overruns 0  frame 0
                TX packets 72174  bytes 61517403 (61.5 MB)
                TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
                device memory 0xf6400000-f641ffff  

        enp7s0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
                ether MACADDRESS:ba  txqueuelen 1000  (Ethernet)
                RX packets 0  bytes 0 (0.0 B)
                RX errors 0  dropped 0  overruns 0  frame 0
                TX packets 0  bytes 0 (0.0 B)
                TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

        enp9s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
                inet 192.168.2.2  netmask 255.255.255.0  broadcast 192.168.2.255
                ether MACADDRESS:ad  txqueuelen 1000  (Ethernet)
                RX packets 46202  bytes 3022408 (3.0 MB)
                RX errors 0  dropped 30259  overruns 0  frame 0
                TX packets 10206  bytes 2482014 (2.4 MB)
                TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

        lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
                inet 127.0.0.1  netmask 255.0.0.0
                loop  txqueuelen 1000  (Local Loopback)
                RX packets 7175  bytes 51455539 (51.4 MB)
                RX errors 0  dropped 0  overruns 0  frame 0
                TX packets 7175  bytes 51455539 (51.4 MB)
                TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

        macvtap0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
                ether MACADDRESS:c3  txqueuelen 500  (Ethernet)
                RX packets 72168  bytes 7205751 (7.2 MB)
                RX errors 4  dropped 4  overruns 0  frame 0
                TX packets 22353  bytes 6380506 (6.3 MB)
                TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

        macvtap1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
                ether MACADDRESS:c7  txqueuelen 500  (Ethernet)
                RX packets 44247  bytes 3085964 (3.0 MB)
                RX errors 0  dropped 0  overruns 0  frame 0
                TX packets 820  bytes 66536 (66.5 KB)
                TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

        virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
                inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
                ether MACADDRESS:bb  txqueuelen 1000  (Ethernet)
                RX packets 0  bytes 0 (0.0 B)
                RX errors 0  dropped 0  overruns 0  frame 0
                TX packets 0  bytes 0 (0.0 B)
                TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

        virbr0-nic: flags=4098<BROADCAST,MULTICAST>  mtu 1500
                ether MACADDRESS:bb  txqueuelen 1000  (Ethernet)
                RX packets 0  bytes 0 (0.0 B)
                RX errors 0  dropped 0  overruns 0  frame 0
                TX packets 0  bytes 0 (0.0 B)
                TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


        ----------------------------------------

        ● 1: lo
               Link File: /lib/systemd/network/99-default.link
            Network File: n/a
                    Type: loopback
                   State: carrier (unmanaged)
                 Address: 127.0.0.1

        ● 2: enp7s0
               Link File: /run/systemd/network/10-netplan-enp7s0.link
            Network File: /run/systemd/network/10-netplan-enp7s0.network
                    Type: ether
                   State: no-carrier (configuring)
                    Path: pci-0000:07:00.0
                  Driver: r8169
                  Vendor: Realtek Semiconductor Co., Ltd.
                   Model: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (TG-3468 Gigabit PCI Express Network Adapter)
              HW Address: MACADDRESS:ba (TP-LINK TECHNOLOGIES CO.,LTD.)
                     DNS: 192.168.2.1
          Search Domains: localdomain

        ● 3: enp9s0
               Link File: /run/systemd/network/10-netplan-enp9s0.link
            Network File: /run/systemd/network/10-netplan-enp9s0.network
                    Type: ether
                   State: routable (configured)
                    Path: pci-0000:09:00.0
                  Driver: r8169
                  Vendor: Realtek Semiconductor Co., Ltd.
                   Model: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (Motherboard (one of many))
              HW Address: MACADDRESS:ad (ASRock Incorporation)
                 Address: 192.168.2.2
                 Gateway: 192.168.2.1 (TP-LINK TECHNOLOGIES CO.,LTD.)
                     DNS: 192.168.2.1
          Search Domains: localdomain
            Connected To: No Site Name on port 192.168.1.120 (No Site Name)

        ● 4: enp4s0f0
               Link File: /run/systemd/network/10-netplan-enp4s0f0.link
            Network File: /run/systemd/network/10-netplan-enp4s0f0.network
                    Type: ether
                   State: routable (configured)
                    Path: pci-0000:04:00.0
                  Driver: igb
                  Vendor: Intel Corporation
                   Model: 82576 Gigabit Network Connection (Gigabit ET Dual Port Server Adapter)
              HW Address: MACADDRESS:74 (Shenzhen Lianrui Electronics Co.,Ltd)
                 Address: 192.168.2.4
                 Gateway: 192.168.2.1
                     DNS: 192.168.2.1
          Search Domains: localdomain
            Connected To: No Site Name on port 192.168.1.120 (No Site Name)

        ● 5: enp4s0f1
               Link File: /run/systemd/network/10-netplan-enp4s0f1.link
            Network File: /run/systemd/network/10-netplan-enp4s0f1.network
                    Type: ether
                   State: routable (configured)
                    Path: pci-0000:04:00.1
                  Driver: igb
                  Vendor: Intel Corporation
                   Model: 82576 Gigabit Network Connection (Gigabit ET Dual Port Server Adapter)
              HW Address: MACADDRESS:75 (Shenzhen Lianrui Electronics Co.,Ltd)
                 Address: 192.168.2.5
                 Gateway: 192.168.2.1 (TP-LINK TECHNOLOGIES CO.,LTD.)
                     DNS: 192.168.2.1
          Search Domains: localdomain
            Connected To: No Site Name on port 192.168.1.120 (No Site Name)

        ● 6: virbr0
               Link File: /lib/systemd/network/99-default.link
            Network File: n/a
                    Type: ether
                   State: no-carrier (unmanaged)
                  Driver: bridge
              HW Address: MACADDRESS:bb
                 Address: 192.168.122.1

        ● 7: virbr0-nic
               Link File: /lib/systemd/network/99-default.link
            Network File: n/a
                    Type: ether
                   State: off (unmanaged)
                  Driver: tun
              HW Address: MACADDRESS:bb

        ● 8: macvtap0
               Link File: /lib/systemd/network/99-default.link
            Network File: n/a
                    Type: ether
                   State: carrier (unmanaged)
                  Driver: macvlan
              HW Address: MACADDRESS:c3

        ● 9: macvtap1
               Link File: /lib/systemd/network/99-default.link
            Network File: n/a
                    Type: ether
                   State: carrier (unmanaged)
                  Driver: macvlan
              HW Address: MACADDRESS:c7

        ----------------------------------------

        Kernel IP routing table
        Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
        0.0.0.0         192.168.2.1     0.0.0.0         UG    0      0        0 enp4s0f1
        0.0.0.0         192.168.2.1     0.0.0.0         UG    0      0        0 enp9s0
        0.0.0.0         192.168.2.1     0.0.0.0         UG    0      0        0 enp4s0f0
        192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 enp4s0f1
        192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 enp9s0
        192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 enp4s0f0
        192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0

        ----------------------------------------

        ethtool enp9s0

        Settings for enp9s0:
            Supported ports: [ TP MII ]
            Supported link modes:   10baseT/Half 10baseT/Full 
                                    100baseT/Half 100baseT/Full 
                                    1000baseT/Half 1000baseT/Full 
            Supported pause frame use: No
            Supports auto-negotiation: Yes
            Supported FEC modes: Not reported
            Advertised link modes:  10baseT/Half 10baseT/Full 
                                    100baseT/Half 100baseT/Full 
                                    1000baseT/Full 
            Advertised pause frame use: Symmetric Receive-only
            Advertised auto-negotiation: Yes
            Advertised FEC modes: Not reported
            Link partner advertised link modes:  10baseT/Half 10baseT/Full 
                                                 100baseT/Half 100baseT/Full 
                                                 1000baseT/Full 
            Link partner advertised pause frame use: No
            Link partner advertised auto-negotiation: Yes
            Link partner advertised FEC modes: Not reported
            Speed: 1000Mb/s
            Duplex: Full
            Port: MII
            PHYAD: 0
            Transceiver: internal
            Auto-negotiation: on
            Supports Wake-on: pumbg
            Wake-on: g
            Current message level: 0x00000033 (51)
                           drv probe ifdown ifup
            Link detected: yes


        ----------------------------------------


        Apr 05 17:53:01 systemname kernel: r8169 0000:09:00.0 enp9s0: renamed from eth1
        Apr 05 17:53:01 systemname kernel: r8169 0000:09:00.0 enp9s0: link down
        Apr 05 17:53:01 systemname kernel: r8169 0000:09:00.0 enp9s0: link down
        Apr 05 17:53:02 systemname systemd-networkd[929]: enp9s0: Link UP
        Apr 05 17:53:05 systemname systemd-networkd[929]: enp9s0: Gained carrier
        Apr 05 17:53:05 systemname systemd-networkd[929]: enp9s0: Configured
        Apr 05 17:53:05 systemname kernel: r8169 0000:09:00.0 enp9s0: link up

        ----------------------------------------

        04:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
        04:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
        07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
        09:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

        ----------------------------------------

        modinfo r8169
        filename:       /lib/modules/4.15.0-91-generic/kernel/drivers/net/ethernet/realtek/r8169.ko
        firmware:       rtl_nic/rtl8107e-2.fw
        firmware:       rtl_nic/rtl8107e-1.fw
        firmware:       rtl_nic/rtl8168h-2.fw
        firmware:       rtl_nic/rtl8168h-1.fw
        firmware:       rtl_nic/rtl8168g-3.fw
        firmware:       rtl_nic/rtl8168g-2.fw
        firmware:       rtl_nic/rtl8106e-2.fw
        firmware:       rtl_nic/rtl8106e-1.fw
        firmware:       rtl_nic/rtl8411-2.fw
        firmware:       rtl_nic/rtl8411-1.fw
        firmware:       rtl_nic/rtl8402-1.fw
        firmware:       rtl_nic/rtl8168f-2.fw
        firmware:       rtl_nic/rtl8168f-1.fw
        firmware:       rtl_nic/rtl8105e-1.fw
        firmware:       rtl_nic/rtl8168e-3.fw
        firmware:       rtl_nic/rtl8168e-2.fw
        firmware:       rtl_nic/rtl8168e-1.fw
        firmware:       rtl_nic/rtl8168d-2.fw
        firmware:       rtl_nic/rtl8168d-1.fw
        version:        2.3LK-NAPI
        license:        GPL
        description:    RealTek RTL-8169 Gigabit Ethernet driver
        author:         Realtek and the Linux r8169 crew <netdev@vger.kernel.org>
        srcversion:     68D71F7AFFEBF88AE55BB02
        alias:          pci:v00000001d00008168sv*sd00002410bc*sc*i*
        alias:          pci:v00001737d00001032sv*sd00000024bc*sc*i*
        alias:          pci:v000016ECd00000116sv*sd*bc*sc*i*
        alias:          pci:v00001259d0000C107sv*sd*bc*sc*i*
        alias:          pci:v00001186d00004302sv*sd*bc*sc*i*
        alias:          pci:v00001186d00004300sv*sd*bc*sc*i*
        alias:          pci:v00001186d00004300sv00001186sd00004B10bc*sc*i*
        alias:          pci:v000010ECd00008169sv*sd*bc*sc*i*
        alias:          pci:v000010FFd00008168sv*sd*bc*sc*i*
        alias:          pci:v000010ECd00008168sv*sd*bc*sc*i*
        alias:          pci:v000010ECd00008167sv*sd*bc*sc*i*
        alias:          pci:v000010ECd00008161sv*sd*bc*sc*i*
        alias:          pci:v000010ECd00008136sv*sd*bc*sc*i*
        alias:          pci:v000010ECd00008129sv*sd*bc*sc*i*
        alias:          pci:v000010ECd00002600sv*sd*bc*sc*i*
        alias:          pci:v000010ECd00002502sv*sd*bc*sc*i*
        depends:        mii
        retpoline:      Y
        intree:         Y
        name:           r8169
        vermagic:       4.15.0-91-generic SMP mod_unload 
        signat:         PKCS#7
        signer:         
        sig_key:        
        sig_hashalgo:   md4
        parm:           use_dac:Enable PCI DAC. Unsafe on 32 bit PCI slot. (int)
        parm:           debug:Debug verbosity level (0=none, ..., 16=all) (int)



        ----------------------------------------


        smod | grep r8169
        r8169                  86016  0
        mii                    16384  1 r8169


        ----------------------------------------


        [    1.825213] r8169 0000:09:00.0 eth1: RTL8168h/8111h at 0x        (ptrval), 70:85:c2:f4:23:ad, XID 14100800 IRQ 54
        [    1.833730] r8169 0000:09:00.0 eth1: jumbo features [frames: 9200 bytes, tx checksumming: ko]
        [    1.916285] r8169 0000:09:00.0 enp9s0: renamed from eth1
        [    2.550164] igb 0000:04:00.1: added PHC on eth1
        [    2.550169] igb 0000:04:00.1: eth1: (PCIe:2.5Gb/s:Width x1) 6c:b3:11:1c:a0:75
        [    2.550175] igb 0000:04:00.1: eth1: PBA No: Unknown


        [    2.550733] igb 0000:04:00.1 enp4s0f1: renamed from eth1


        [    1.916285] r8169 0000:09:00.0 enp9s0: renamed from eth1
        [    5.694631] r8169 0000:09:00.0 enp9s0: link down
        [    5.694633] r8169 0000:09:00.0 enp9s0: link down
        [    5.694711] IPv6: ADDRCONF(NETDEV_UP): enp9s0: link is not ready
        [    8.803624] r8169 0000:09:00.0 enp9s0: link up
        [    8.803628] IPv6: ADDRCONF(NETDEV_CHANGE): enp9s0: link becomes ready


        [    1.812022] r8169 0000:07:00.0 eth0: RTL8168e/8111e at 0x        (ptrval), 7c:8b:ca:03:76:ba, XID 0c200000 IRQ 53
        [    1.812023] r8169 0000:07:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
        [    1.868198] r8169 0000:07:00.0 enp7s0: renamed from eth0
        [    2.334161] igb 0000:04:00.0: added PHC on eth0
        [    2.334165] igb 0000:04:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 6c:b3:11:1c:a0:74
        [    2.334170] igb 0000:04:00.0: eth0: PBA No: Unknown
        [    2.596602] igb 0000:04:00.0 enp4s0f0: renamed from eth0



        [    1.868198] r8169 0000:07:00.0 enp7s0: renamed from eth0
        [    5.867125] r8169 0000:07:00.0 enp7s0: link down
        [    5.867187] IPv6: ADDRCONF(NETDEV_UP): enp7s0: link is not ready


        [    2.550733] igb 0000:04:00.1 enp4s0f1: renamed from eth1
        [    5.391859] IPv6: ADDRCONF(NETDEV_UP): enp4s0f1: link is not ready
        [    7.712704] igb 0000:04:00.1 enp4s0f1: igb: enp4s0f1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
        [    7.928351] IPv6: ADDRCONF(NETDEV_CHANGE): enp4s0f1: link becomes ready


        [    2.596602] igb 0000:04:00.0 enp4s0f0: renamed from eth0
        [    5.639862] IPv6: ADDRCONF(NETDEV_UP): enp4s0f0: link is not ready
        [   13.868125] igb 0000:04:00.0 enp4s0f0: igb: enp4s0f0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX
        [   13.868266] igb 0000:04:00.0 enp4s0f0: Link Speed was downgraded by SmartSpeed
        [   13.868513] IPv6: ADDRCONF(NETDEV_CHANGE): enp4s0f0: link becomes ready



        ----------------------------------------

        YAML FILES

        01-netcfg.yaml

        network:
          version: 2
          renderer: networkd
          ethernets:
            enp9s0:
              match:
                macaddress: MACADDRESS:ad
              addresses: [192.168.2.2/24]
              gateway4: 192.168.2.1
              nameservers:
                addresses: [192.168.2.1,208.67.222.222,208.67.220.220]
                search: [localdomain]

        02-netcfg.yaml

        network:
          version: 2
          renderer: networkd
          ethernets:
            enp7s0:
              match:
                macaddress: MACADDRESS:ba
              addresses: [192.168.2.3/24]
              gateway4: 192.168.2.1
              nameservers:
                addresses: [192.168.2.1,208.67.222.222,208.67.220.220]
                search: [localdomain]

        03-netcfg.yaml

        network:
          version: 2
          renderer: networkd
          ethernets:
            enp4s0f0:
              match:
                macaddress: MACADDRESS:74
              addresses: [192.168.2.4/24]
              gateway4: 192.168.2.1
              nameservers:
                addresses: [192.168.2.1,208.67.222.222,208.67.220.220]
                search: [localdomain]

        04-netcfg.yaml

        network:
          version: 2
          renderer: networkd
          ethernets:
            enp4s0f1:
              match:
                macaddress: MACADDRESS:75
              addresses: [192.168.2.5/24]
              gateway4: 192.168.2.1
              nameservers:
                addresses: [192.168.2.1,208.67.222.222,208.67.220.220]
                search: [localdomain]

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.