New Dell 7090 Tower - Installed Ubuntu 20.10, 21.04 and 21.10 - Getting network packet loss until usb thumb drive plugged in

Question

I recently got a new Dell Optiplex 7090 Machine, i9- 10900, 64GB Ram

I attempted to install Ubuntu 20.04 (Server, not using GUI so only using the server version) on it but after the first screen where you select install Ubuntu the display goes blank/loses signal. I assume the hardware is too new for 20.04.

I installed 20.10 (or maybe it was 21.04) and it installed fine, no issues, smooth as expected. However after doing basic setup on it for 15mins or so my SSH connection became slow/got dropped. Investigating I noticed heavy packet loss - it looked like an IP Conflict on the network. I checked & changed IP but no difference.

It's using onboard Gbit LAN plugged direct into the router. I swapped the network cable with my other PC next to it that had no issue (this meant it was also using a different port on the router as well). I tried setting a static IP I tried reinstalling from scratch, tried different versions of Ubuntu 20.04/21.04/21.10(where i am on now). I tried CentOS 8 - this worked fine, no network packet loss, but I prefer to use Ubuntu so want to get it working.

In the end i figured out that if I plug in a certain USB Memory stick into the front USB3 port the packet loss stops and the network connection becomes stable. Now if I were you I wouldn't believe me either, but I promise you this makes the network connection stable. I can't for the life of me figure out how this could make any difference. But this has persisted across different re-installations. I install Ubuntu from the USB thumb drive, i remove it, I use it for 15mins or so and then packet loss starts happening. I then plug in the USB drive and it's now 100% stable (including pinging overnight).

So I'm at a loss, would love to figure this out if anyone has any ideas/tips to try?

lspci -v
00:00.0 Host bridge: Intel Corporation Device 9b33 (rev 05)
    Subsystem: Dell Device 0a52
    Flags: bus master, fast devsel, latency 0
    Capabilities: <access denied>
    Kernel driver in use: skl_uncore

00:02.0 VGA compatible controller: Intel Corporation CometLake-S GT2 [UHD Graphics 630] (rev 05) (prog-if 00 [VGA controller])
    Subsystem: Dell CometLake-S GT2 [UHD Graphics 630]
    Flags: bus master, fast devsel, latency 0, IRQ 148
    Memory at 6000000000 (64-bit, non-prefetchable) [size=16M]
    Memory at 4000000000 (64-bit, prefetchable) [size=256M]
    I/O ports at 3000 [size=64]
    Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
    Capabilities: <access denied>
    Kernel driver in use: i915
    Kernel modules: i915

00:14.0 USB controller: Intel Corporation Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller (rev 11) (prog-if 30 [XHCI])
    Subsystem: Dell Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller
    Flags: bus master, medium devsel, latency 0, IRQ 125
    Memory at 6001100000 (64-bit, non-prefetchable) [size=64K]
    Capabilities: <access denied>
    Kernel driver in use: xhci_hcd
    Kernel modules: xhci_pci

00:14.2 RAM memory: Intel Corporation Tiger Lake-H Shared SRAM (rev 11)
    Subsystem: Dell Tiger Lake-H Shared SRAM
    Flags: fast devsel
    Memory at 6001114000 (64-bit, non-prefetchable) [disabled] [size=16K]
    Memory at 600111b000 (64-bit, non-prefetchable) [disabled] [size=4K]
    Capabilities: <access denied>

00:15.0 Serial bus controller [0c80]: Intel Corporation Tiger Lake-H Serial IO I2C Controller #0 (rev 11)
    Subsystem: Dell Tiger Lake-H Serial IO I2C Controller
    Flags: bus master, fast devsel, latency 0, IRQ 27
    Memory at 4010000000 (64-bit, non-prefetchable) [virtual] [size=4K]
    Capabilities: <access denied>
    Kernel driver in use: intel-lpss
    Kernel modules: intel_lpss_pci

00:16.0 Communication controller: Intel Corporation Tiger Lake-H Management Engine Interface (rev 11)
    Subsystem: Dell Tiger Lake-H Management Engine Interface
    Flags: bus master, fast devsel, latency 0, IRQ 149
    Memory at 6001119000 (64-bit, non-prefetchable) [size=4K]
    Capabilities: <access denied>
    Kernel driver in use: mei_me
    Kernel modules: mei_me

00:17.0 SATA controller: Intel Corporation Device 43d2 (rev 11) (prog-if 01 [AHCI 1.0])
    Subsystem: Dell Device 0a52
    Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 127
    Memory at 6e920000 (32-bit, non-prefetchable) [size=8K]
    Memory at 6e924000 (32-bit, non-prefetchable) [size=256]
    I/O ports at 3080 [size=8]
    I/O ports at 3088 [size=4]
    I/O ports at 3060 [size=32]
    Memory at 6e923000 (32-bit, non-prefetchable) [size=2K]
    Capabilities: <access denied>
    Kernel driver in use: ahci
    Kernel modules: ahci

00:1b.0 PCI bridge: Intel Corporation Device 43c4 (rev 11) (prog-if 00 [Normal decode])
    Flags: bus master, fast devsel, latency 0, IRQ 122
    Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
    I/O behind bridge: [disabled]
    Memory behind bridge: 6e800000-6e8fffff [size=1M]
    Prefetchable memory behind bridge: [disabled]
    Capabilities: <access denied>
    Kernel driver in use: pcieport

00:1c.0 PCI bridge: Intel Corporation Device 43bd (rev 11) (prog-if 00 [Normal decode])
    Flags: bus master, fast devsel, latency 0, IRQ 123
    Bus: primary=00, secondary=02, subordinate=03, sec-latency=0
    I/O behind bridge: [disabled]
    Memory behind bridge: [disabled]
    Prefetchable memory behind bridge: [disabled]
    Capabilities: <access denied>
    Kernel driver in use: pcieport

00:1f.0 ISA bridge: Intel Corporation Device 4384 (rev 11)
    Subsystem: Dell Device 0a52
    Flags: bus master, fast devsel, latency 0

00:1f.3 Audio device: Intel Corporation Device f0c8 (rev 11)
    Subsystem: Dell Device 0a52
    Flags: bus master, fast devsel, latency 64, IRQ 150
    Memory at 6001110000 (64-bit, non-prefetchable) [size=16K]
    Memory at 6001000000 (64-bit, non-prefetchable) [size=1M]
    Capabilities: <access denied>
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel

00:1f.4 SMBus: Intel Corporation Tiger Lake-H SMBus Controller (rev 11)
    Subsystem: Dell Tiger Lake-H SMBus Controller
    Flags: medium devsel, IRQ 16
    Memory at 6001118000 (64-bit, non-prefetchable) [size=256]
    I/O ports at efa0 [size=32]
    Kernel driver in use: i801_smbus
    Kernel modules: i2c_i801

00:1f.5 Serial bus controller [0c80]: Intel Corporation Tiger Lake-H SPI Controller (rev 11)
    Subsystem: Dell Tiger Lake-H SPI Controller
    Flags: fast devsel
    Memory at 6e922000 (32-bit, non-prefetchable) [size=4K]

00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (14) I219-LM (rev 11)
    Subsystem: Dell Ethernet Connection (14) I219-LM
    Flags: bus master, fast devsel, latency 0, IRQ 124
    Memory at 6e900000 (32-bit, non-prefetchable) [size=128K]
    Capabilities: <access denied>
    Kernel driver in use: e1000e
    Kernel modules: e1000e

01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/980PRO (prog-if 02 [NVM Express])
    Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/980PRO
    Flags: bus master, fast devsel, latency 0, IRQ 16, NUMA node 0
    Memory at 6e800000 (64-bit, non-prefetchable) [size=16K]
    Capabilities: <access denied>
    Kernel driver in use: nvme
    Kernel modules: nvme

02:00.0 PCI bridge: Texas Instruments XIO2001 PCI Express-to-PCI Bridge (prog-if 00 [Normal decode])
    Flags: bus master, fast devsel, latency 0
    Bus: primary=02, secondary=03, subordinate=03, sec-latency=0
    I/O behind bridge: [disabled]
    Memory behind bridge: [disabled]
    Prefetchable memory behind bridge: [disabled]
    Capabilities: <access denied>

Answer 1

An MTU of 1500 is the default and should be good in most situations. That said, some networks need something smaller, such as 1492, and do not communicate this as part of the initial negotiation. It would be a good idea to do some testing to see whether the MTU setting is the reason for your excessive packet loss.

Here's how:

0. If you have made any changes to your /etc/dhcp/dhclient.conf file, remove them and reboot the server

1. Use ping to check for packet loss while changing the MTU setting:

   ping -c 4 -M do -s 1472 151.101.193.69
   

   Definitions:

   Option	Definition
   -c 4	Count ⇢ The number of times to iterate
   -M do	Set path MTU discovery strategy ⇢ do will not allow fragmentation at any level
   -s 1472	Packet size in bits (1472 + 28 overhead = 1500)
   151.101.193.69	The IP address for AskUbuntu. Feel free to use any address, but avoid Google-owned services as they will not always respond to ping requests.

   You may see output like this:

   ping -c 4 -M do -s 1472 151.101.193.69
   PING 151.101.193.69 (151.101.193.69) 1472(1500) bytes of data.
   From 192.168.0.1 icmp_seq=1 Frag needed and DF set (mtu = 1454)
   ping: local error: message too long, mtu=1454
   ping: local error: message too long, mtu=1454
   ping: local error: message too long, mtu=1454
   
   --- 151.101.193.69 ping statistics ---
   4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 3074ms
   

2. Play with the -s value until you find no fragmentation and no message too long error. For example:

   ping -c 4 -M do -s 1200 151.101.193.69
   PING 151.101.193.69 (151.101.193.69) 1200(1228) bytes of data.
   1208 bytes from 151.101.193.69: icmp_seq=1 ttl=59 time=5.52 ms
   1208 bytes from 151.101.193.69: icmp_seq=2 ttl=59 time=5.72 ms
   1208 bytes from 151.101.193.69: icmp_seq=3 ttl=59 time=5.64 ms
   1208 bytes from 151.101.193.69: icmp_seq=4 ttl=59 time=5.68 ms
   
   --- 151.101.193.69 ping statistics ---
   4 packets transmitted, 4 received, 0% packet loss, time 3005ms
   rtt min/avg/max/mdev = 5.517/5.640/5.720/0.076 ms
   

   For this example, I found that 1200, which is an MTU of 1228 results in zero packet loss.

   Note: Ideally you would start at 1472 and work your way down in increments of 10 or so. If the MTU value is too low, your transfer speed will suffer.

3. Set the appropriate MTU value for your network in your /etc/dhcp/dhclient.conf file:

   default interface-mtu 1228;
   supersede interface-mtu 1228;
   

   _{Note: Be sure to change 1228 to the appropriate MTU size for your network.}

   If you have multiple interfaces, you can specify which one(s) should have what MTU value:

   interface "enp0s31f6" {
       default interface-mtu 1228;
       supersede interface-mtu 1228;
   }
   

4. Restart networking and ensure the interface is up:

   sudo service networking restart
   sudo ifup enp0s31f6
   

5. Verify the MTU value:

   sudo ifconfig | grep mtu
   

   Which may give you something like:

   docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
   enp0s31f6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1228
   lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
   veth43316c1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
   veth80786dc: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
   vethf40bd74: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
   

With this, hopefully your packet loss will be a thing of the past 👍🏻