PCI BAR "failed to assign" errors with SANLink2 Thunderbolt device

Question

I am trying to connect a Promise SANLink2 Thunderbolt-2-to-10-GbE-Ethernet device to a Lenovo ThinkPad running Ubuntu 16.04. The system does not recognize the dual Ethernet interfaces on the device. If I look at dmesg after plugging in the device, I see the following:

[  169.822146] pci 0000:06:00.0: [8086:1567] type 01 class 0x060400
[  169.822534] pci 0000:06:00.0: supports D1 D2
[  169.822536] pci 0000:06:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[  169.830152] pci 0000:07:00.0: [8086:1567] type 01 class 0x060400
[  169.830362] pci 0000:07:00.0: supports D1 D2
[  169.830364] pci 0000:07:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[  169.830522] pci 0000:07:03.0: [8086:1567] type 01 class 0x060400
[  169.830729] pci 0000:07:03.0: supports D1 D2
[  169.830730] pci 0000:07:03.0: PME# supported from D0 D1 D2 D3hot D3cold
[  169.830841] pci 0000:07:04.0: [8086:1567] type 01 class 0x060400
[  169.831018] pci 0000:07:04.0: supports D1 D2
[  169.831019] pci 0000:07:04.0: PME# supported from D0 D1 D2 D3hot D3cold
[  169.831117] pci 0000:07:05.0: [8086:1567] type 01 class 0x060400
[  169.831350] pci 0000:07:05.0: supports D1 D2
[  169.831351] pci 0000:07:05.0: PME# supported from D0 D1 D2 D3hot D3cold
[  169.831491] pci 0000:07:06.0: [8086:1567] type 01 class 0x060400
[  169.831609] pci 0000:07:06.0: supports D1 D2
[  169.831610] pci 0000:07:06.0: PME# supported from D0 D1 D2 D3hot D3cold
[  169.831753] pci 0000:06:00.0: PCI bridge to [bus 07-3f]
[  169.831768] pci 0000:06:00.0:   bridge window [mem 0xb8000000-0xce0fffff]
[  169.831785] pci 0000:06:00.0:   bridge window [mem 0xd0000000-0xf1ffffff 64bit pref]
[  169.831915] pci 0000:08:00.0: [8086:1566] type 00 class 0x088000
[  169.831966] pci 0000:08:00.0: reg 0x10: [mem 0xce000000-0xce03ffff]
[  169.831982] pci 0000:08:00.0: reg 0x14: [mem 0xce040000-0xce040fff]
[  169.832327] pci 0000:08:00.0: supports D1 D2
[  169.832328] pci 0000:08:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[  169.832498] pci 0000:07:00.0: PCI bridge to [bus 08]
[  169.832525] pci 0000:07:00.0:   bridge window [mem 0xce000000-0xce0fffff]
[  169.832648] pci 0000:07:03.0: PCI bridge to [bus 09-34]
[  169.832675] pci 0000:07:03.0:   bridge window [mem 0xb8000000-0xc5ffffff]
[  169.832693] pci 0000:07:03.0:   bridge window [mem 0xd0000000-0xe9ffffff 64bit pref]
[  169.832833] pci 0000:07:04.0: PCI bridge to [bus 35-3d]
[  169.832860] pci 0000:07:04.0:   bridge window [mem 0xc6000000-0xcdffffff]
[  169.832877] pci 0000:07:04.0:   bridge window [mem 0xea000000-0xf1ffffff 64bit pref]
[  169.833017] pci 0000:07:05.0: PCI bridge to [bus 3e]
[  169.833181] pci 0000:07:06.0: PCI bridge to [bus 3f]
[  169.833301] pci_bus 0000:07: Allocating resources
[  169.833365] pci 0000:07:00.0: bridge window [io  0x1000-0x0fff] to [bus 08] add_size 1000
[  169.833369] pci 0000:07:00.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 08] add_size 200000 add_align 100000
[  169.833395] pci 0000:07:03.0: bridge window [io  0x1000-0x0fff] to [bus 09-34] add_size 1000
[  169.833421] pci 0000:07:04.0: bridge window [io  0x1000-0x0fff] to [bus 35-3d] add_size 1000
[  169.833446] pci 0000:07:05.0: bridge window [io  0x1000-0x0fff] to [bus 3e] add_size 1000
[  169.833448] pci 0000:07:05.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 3e] add_size 200000 add_align 100000
[  169.833450] pci 0000:07:05.0: bridge window [mem 0x00100000-0x000fffff] to [bus 3e] add_size 200000 add_align 100000
[  169.833475] pci 0000:07:06.0: bridge window [io  0x1000-0x0fff] to [bus 3f] add_size 1000
[  169.833477] pci 0000:07:06.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 3f] add_size 200000 add_align 100000
[  169.833478] pci 0000:07:06.0: bridge window [mem 0x00100000-0x000fffff] to [bus 3f] add_size 200000 add_align 100000
[  169.833505] pci 0000:07:00.0: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[  169.833507] pci 0000:07:03.0: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[  169.833509] pci 0000:07:04.0: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[  169.833511] pci 0000:07:05.0: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[  169.833514] pci 0000:07:06.0: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[  169.833516] pci 0000:06:00.0: bridge window [io  0x1000-0x0fff] to [bus 07-3f] add_size 5000
[  169.833519] pci 0000:06:00.0: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 5000 min_align 1000
[  169.833521] pci 0000:06:00.0: res[13]=[io  0x1000-0x5fff] res_to_dev_res add_size 5000 min_align 1000
[  169.833525] pci 0000:06:00.0: BAR 13: assigned [io  0x6000-0xafff]
[  169.833532] pci 0000:07:00.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[  169.833534] pci 0000:07:00.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[  169.833537] pci 0000:07:05.0: res[14]=[mem 0x00100000-0x000fffff] res_to_dev_res add_size 200000 min_align 100000
[  169.833539] pci 0000:07:05.0: res[14]=[mem 0x00100000-0x002fffff] res_to_dev_res add_size 200000 min_align 100000
[  169.833542] pci 0000:07:05.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[  169.833544] pci 0000:07:05.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[  169.833547] pci 0000:07:06.0: res[14]=[mem 0x00100000-0x000fffff] res_to_dev_res add_size 200000 min_align 100000
[  169.833549] pci 0000:07:06.0: res[14]=[mem 0x00100000-0x002fffff] res_to_dev_res add_size 200000 min_align 100000
[  169.833552] pci 0000:07:06.0: res[15]=[mem 0x00100000-0x000fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[  169.833554] pci 0000:07:06.0: res[15]=[mem 0x00100000-0x002fffff 64bit pref] res_to_dev_res add_size 200000 min_align 100000
[  169.833556] pci 0000:07:00.0: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[  169.833559] pci 0000:07:00.0: res[13]=[io  0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
[  169.833561] pci 0000:07:03.0: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[  169.833563] pci 0000:07:03.0: res[13]=[io  0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
[  169.833565] pci 0000:07:04.0: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[  169.833568] pci 0000:07:04.0: res[13]=[io  0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
[  169.833570] pci 0000:07:05.0: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[  169.833573] pci 0000:07:05.0: res[13]=[io  0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
[  169.833575] pci 0000:07:06.0: res[13]=[io  0x1000-0x0fff] res_to_dev_res add_size 1000 min_align 1000
[  169.833577] pci 0000:07:06.0: res[13]=[io  0x1000-0x1fff] res_to_dev_res add_size 1000 min_align 1000
[  169.833581] pci 0000:07:00.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[  169.833583] pci 0000:07:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[  169.833585] pci 0000:07:05.0: BAR 14: no space for [mem size 0x00200000]
[  169.833587] pci 0000:07:05.0: BAR 14: failed to assign [mem size 0x00200000]
[  169.833590] pci 0000:07:05.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[  169.833593] pci 0000:07:05.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[  169.833595] pci 0000:07:06.0: BAR 14: no space for [mem size 0x00200000]
[  169.833597] pci 0000:07:06.0: BAR 14: failed to assign [mem size 0x00200000]
[  169.833600] pci 0000:07:06.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[  169.833602] pci 0000:07:06.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[  169.833605] pci 0000:07:00.0: BAR 13: assigned [io  0x6000-0x6fff]
[  169.833607] pci 0000:07:03.0: BAR 13: assigned [io  0x7000-0x7fff]
[  169.833609] pci 0000:07:04.0: BAR 13: assigned [io  0x8000-0x8fff]
[  169.833611] pci 0000:07:05.0: BAR 13: assigned [io  0x9000-0x9fff]
[  169.833614] pci 0000:07:06.0: BAR 13: assigned [io  0xa000-0xafff]
[  169.833618] pci 0000:07:06.0: BAR 14: no space for [mem size 0x00200000]
[  169.833621] pci 0000:07:06.0: BAR 14: failed to assign [mem size 0x00200000]
[  169.833624] pci 0000:07:06.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[  169.833626] pci 0000:07:06.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[  169.833629] pci 0000:07:05.0: BAR 14: no space for [mem size 0x00200000]
[  169.833631] pci 0000:07:05.0: BAR 14: failed to assign [mem size 0x00200000]
[  169.833634] pci 0000:07:05.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[  169.833636] pci 0000:07:05.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[  169.833640] pci 0000:07:00.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[  169.833642] pci 0000:07:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[  169.833645] pci 0000:07:00.0: PCI bridge to [bus 08]
[  169.833652] pci 0000:07:00.0:   bridge window [io  0x6000-0x6fff]
[  169.833668] pci 0000:07:00.0:   bridge window [mem 0xce000000-0xce0fffff]
[  169.833699] pci 0000:07:03.0: PCI bridge to [bus 09-34]
[  169.833706] pci 0000:07:03.0:   bridge window [io  0x7000-0x7fff]
[  169.833722] pci 0000:07:03.0:   bridge window [mem 0xb8000000-0xc5ffffff]
[  169.833733] pci 0000:07:03.0:   bridge window [mem 0xd0000000-0xe9ffffff 64bit pref]
[  169.833754] pci 0000:07:04.0: PCI bridge to [bus 35-3d]
[  169.833760] pci 0000:07:04.0:   bridge window [io  0x8000-0x8fff]
[  169.833776] pci 0000:07:04.0:   bridge window [mem 0xc6000000-0xcdffffff]
[  169.833787] pci 0000:07:04.0:   bridge window [mem 0xea000000-0xf1ffffff 64bit pref]
[  169.833807] pci 0000:07:05.0: PCI bridge to [bus 3e]
[  169.833813] pci 0000:07:05.0:   bridge window [io  0x9000-0x9fff]
[  169.833889] pci 0000:07:06.0: PCI bridge to [bus 3f]
[  169.833896] pci 0000:07:06.0:   bridge window [io  0xa000-0xafff]
[  169.833937] pci 0000:06:00.0: PCI bridge to [bus 07-3f]
[  169.833943] pci 0000:06:00.0:   bridge window [io  0x6000-0xafff]
[  169.833968] pci 0000:06:00.0:   bridge window [mem 0xb8000000-0xce0fffff]
[  169.833978] pci 0000:06:00.0:   bridge window [mem 0xd0000000-0xf1ffffff 64bit pref]
[  169.834050] pcieport 0000:06:00.0: enabling device (0006 -> 0007)
[  169.834409] pcieport 0000:07:00.0: enabling device (0006 -> 0007)
[  169.834672] pcieport 0000:07:03.0: enabling device (0006 -> 0007)
[  169.834952] pcieport 0000:07:04.0: enabling device (0006 -> 0007)
[  169.835555] pcieport 0000:07:05.0: enabling device (0004 -> 0005)
[  169.835778] pcieport 0000:07:06.0: enabling device (0004 -> 0005)

Note the region that appears to be an issue:

[  169.833581] pci 0000:07:00.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[  169.833583] pci 0000:07:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[  169.833585] pci 0000:07:05.0: BAR 14: no space for [mem size 0x00200000]
[  169.833587] pci 0000:07:05.0: BAR 14: failed to assign [mem size 0x00200000]
[  169.833590] pci 0000:07:05.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[  169.833593] pci 0000:07:05.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[  169.833595] pci 0000:07:06.0: BAR 14: no space for [mem size 0x00200000]
[  169.833597] pci 0000:07:06.0: BAR 14: failed to assign [mem size 0x00200000]
[  169.833600] pci 0000:07:06.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[  169.833602] pci 0000:07:06.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[  169.833605] pci 0000:07:00.0: BAR 13: assigned [io  0x6000-0x6fff]
[  169.833607] pci 0000:07:03.0: BAR 13: assigned [io  0x7000-0x7fff]
[  169.833609] pci 0000:07:04.0: BAR 13: assigned [io  0x8000-0x8fff]
[  169.833611] pci 0000:07:05.0: BAR 13: assigned [io  0x9000-0x9fff]
[  169.833614] pci 0000:07:06.0: BAR 13: assigned [io  0xa000-0xafff]
[  169.833618] pci 0000:07:06.0: BAR 14: no space for [mem size 0x00200000]
[  169.833621] pci 0000:07:06.0: BAR 14: failed to assign [mem size 0x00200000]
[  169.833624] pci 0000:07:06.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[  169.833626] pci 0000:07:06.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[  169.833629] pci 0000:07:05.0: BAR 14: no space for [mem size 0x00200000]
[  169.833631] pci 0000:07:05.0: BAR 14: failed to assign [mem size 0x00200000]
[  169.833634] pci 0000:07:05.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[  169.833636] pci 0000:07:05.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]
[  169.833640] pci 0000:07:00.0: BAR 15: no space for [mem size 0x00200000 64bit pref]
[  169.833642] pci 0000:07:00.0: BAR 15: failed to assign [mem size 0x00200000 64bit pref]

lspci shows the PCI bridge devices that appear when I plug in the SANLink2, but it doesn't show anything attached to those bridges. You can see the full lspci -vvxxxx output here.

I've found mentions of similar issues on other distro bug trackers, but no solutions that help my problem. I've tried the latest mainline kernel (v4.7) to see if any recent bugfixes had an effect, but I see the same behavior.

Answer 1

If I could comment, this would be a comment - because I do not yet have a definitive solution. I am suffering this on Thunderbolt 3 with Titan Ridge add-in cards. It only seems to affect my eGPU - my SANLink3 N1 and Dell Thunderbolt 3 SSD appear unaffected. Either their BARs are assigned, or they are not affected by them failing to be assigned. I have done things to help, but have not fully solved it yet. My external R9 Nano has a BAR like yours, possibly with the exact size and type, that is failing: [mem size 0x00200000 64bit pref]

Some things that I have found may help:

• Install BIOS update if available
• Update NVM Thunderbolt firmware if available (https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/thunderbolt.rst)
• Intel VT-d enabled in BIOS
• Above 4G decoding enabled in BIOS
• In the GRUB you can set kernel parameters
• intel_iommu=on or intel_iommu=igfx_only if that breaks your video
• pci=nocrs,realloc (nocrs allows it to ignore more constraints placed by the BIOS)
• Maybe try pci=noacpi (might break things, or might not)
• Find more options here: https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
• Try the latest kernel - Linux 4.19 (stable) or Linux 4.20-rc2 (bleeding edge)

With Titan Ridge, pci=assign-busses can help a lot, particularly if using the Thunderbolt controller on an unsupported system (such as AMD), but this breaks any Thunderbolt implementation prior to Titan Ridge - so do not use this one.

I do not know if Thunderbolt 2 had security levels or not. But as a last resort, you can enable Thunderbolt boot support and Thunderbolt Option ROM in BIOS, and disable security levels if applicable. Then attach the SANLink to the computer before you power it on, and hopefully the BIOS will allocate it (in this case, remove the kernel parameters you were experimenting with for Linux). This is not the way we want to move forward - everything should be dealt with in the OS where the code can be easily updated, audited and debugged. But for now, the BIOS will continue to plague us.

If you wish to use the 4.20-rc2 kernel right now, you will have to compile your own - as once again, Ubuntu is failing mainline builds and nobody is doing anything about it. http://kernel.ubuntu.com/~kernel-ppa/mainline/

Best of luck! I am aware this is two years and three months ago, so you might not even have the computer anymore. But it has 766 views, and perhaps other people would be interested in some semblance of a solution.