Ubuntu sees only one of two nvme drives that are visible to Bios and Windows 10: PCI problem

Question

I upgraded a MSI GT73VR laptop which has two NVME slots (and one SATA slot). I discovered that Ubuntu 20.04 and 18.04 can only see one of the drives (in the first slot). Switching the M.2 cards allows whichever drive is in the first slot to be seen. The BIOS and Windows 10 can both see the drives and Windows can access both. One drive is a ADATA SX8100NP, currently visible, the other is an Intel 660

The BIOS is set to AHCI access mode. The problem may more Linux-related than Ubuntu-related per se as when I tried Manjaro and OpenSUSE Installation isos from a USB they could also only see only one NVME drive: /dev/nvme0n1.

$ ls /dev/nvme*
/dev/nvme0    /dev/nvme0n1p1   /dev/nvme0n1p11  /dev/nvme0n1p13  /dev/nvme0n1p2  /dev/nvme0n1p4  /dev/nvme0n1p6  /dev/nvme0n1p8
/dev/nvme0n1  /dev/nvme0n1p10  /dev/nvme0n1p12  /dev/nvme0n1p14  /dev/nvme0n1p3  /dev/nvme0n1p5  /dev/nvme0n1p7  /dev/nvme0n1p9

Attempting to debug I see that the problem is related to the PCIE bus, as both nvme drives are visible on the PCI bus.

$ lspci |grep memory
02:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. Device 5762 (rev 01)
3f:00.0 Non-Volatile memory controller: Intel Corporation Device f1a8 (rev 03)

More details of the invisible nvme drive:

~$ sudo lspci -v -s 3f:00.0 
3f:00.0 Non-Volatile memory controller: Intel Corporation Device f1a8 (rev 03) (prog-if 02 [NVM Express])
        Subsystem: Intel Corporation Device 390d
        Flags: bus master, fast devsel, latency 0, IRQ 255, NUMA node 0
        Memory at <ignored> (64-bit, non-prefetchable)
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable- Count=1/8 Maskable+ 64bit+
        Capabilities: [70] Express Endpoint, MSI 00
        Capabilities: [b0] MSI-X: Enable- Count=16 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [158] #19
        Capabilities: [178] Latency Tolerance Reporting
        Capabilities: [180] L1 PM Substates
        Kernel modules: nvme

Now, this led me to check dmesg:

[    0.646094] pci 0000:3f:00.0: [8086:f1a8] type 00 class 0x010802
[    0.646529] pci 0000:3f:00.0: reg 0x10: [mem 0xdc200000-0xdc203fff 64bit]
[    0.648075] pci 0000:3f:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x4 link at 0000:07:04.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[    0.815140] pci 0000:3f:00.0: can't claim BAR 0 [mem 0xdc200000-0xdc203fff 64bit]: no compatible bridge window
[    0.846800] pci 0000:3f:00.0: BAR 0: no space for [mem size 0x00004000 64bit]
[    0.846801] pci 0000:3f:00.0: BAR 0: trying firmware assignment [mem 0xdc200000-0xdc203fff 64bit]
[    0.846803] pci 0000:3f:00.0: BAR 0: [mem 0xdc200000-0xdc203fff 64bit] conflicts with PCI Bus 0000:00 [mem 0x40000000-0xdfffffff window]
[    0.846804] pci 0000:3f:00.0: BAR 0: failed to assign [mem size 0x00004000 64bit]
[    1.978833] pci 0000:3f:00.0: Adding to iommu group 21

Seems to be due to an inability to assign memory to the PCI device, possbily because of a bus conflict.

Anyone know how to resolve this? The problem exists on kernel 5.4.0.26 as well as on mainline kernel 5.9.12 that I temporarily installed, and whatever kernels are on the other linux installation media I tried.

The parameters the kernel were booted with included changing the max latency of nvme drives, but that had no effect either way:

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.4.0-26-generic root=UUID=f91c3fe3-d5c2-45c7-bb74-b2200f7d4057 ro quiet splash intel_iommu=on nvme_core.default_ps_max_latency_us=5500

Any other kernel paramaters to try? Thanks!

Answer 1

I managed to solve the problem by adding the kernel parameter pci=assign-busses

Now we dmesg shows:

$ sudo dmesg |grep 0c:00
[    0.645602] pci 0000:0c:00.0: [8086:f1a8] type 00 class 0x010802
[    0.645831] pci 0000:0c:00.0: reg 0x10: [mem 0xdc200000-0xdc203fff 64bit]
[    2.012831] pci 0000:0c:00.0: Adding to iommu group 22
[    2.435258] nvme nvme1: pci function 0000:0c:00.0

and nvme list shows

$ sudo nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev  
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     2K482919EDEF         ADATA SX8100NP                           1           2.05  TB /   2.05  TB    512   B +  0 B   VB421D57
/dev/nvme1n1     PHNH920600M92P0C     INTEL SSDPEKNW020T8                      1           2.05  TB /   2.05  TB    512   B +  0 B   004C 

Other messages in dmesg during the problem contained "busn ... hidden behind bridge..." I sought out a parameter that would remap the PCI devices ignoring the firmware. From the documentation:

"The `pci=assign-busses' Argument
This tells the kernel to always assign all PCI bus numbers, overriding whatever the firmware may have done."

This does the trick no only for Ubuntu Bionic and Focal but for all linux distros that I tried: OpenSuse Tumbleweed, Debian, and Manjaro. Hopefully someone else will find this helpful.