1

I'm trying to do GPU passthrough for GTX 1060 on Ryzen 7 2700x.

I followed https://mathiashueber.com/windows-virtual-machine-gpu-passthrough-ubuntu/ but I couldn't blacklist the nvidia driver as the host's GPU is also NVIDIA, so I use quirks mentioned below.

When I start Ubuntu 19 in the host machine, I get this in my lspci -nnv:

07:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116 [GeForce GTX 1660] [10de:2184] (rev a1) (prog-if 00 [VGA controller])
    Subsystem: NVIDIA Corporation TU116 [GeForce GTX 1660] [10de:1324]
    Flags: bus master, fast devsel, latency 0, IRQ 11
    Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
    Memory at d0000000 (64-bit, prefetchable) [size=256M]
    Memory at e0000000 (64-bit, prefetchable) [size=32M]
    I/O ports at f000 [size=128]
    Expansion ROM at 000c0000 [disabled] [size=128K]
    Capabilities: <access denied>
    Kernel driver in use: vfio-pci
    Kernel modules: nvidiafb, nouveau

07:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aeb] (rev a1)
    Subsystem: NVIDIA Corporation Device [10de:1324]
    Flags: bus master, fast devsel, latency 0, IRQ 81
    Memory at f7080000 (32-bit, non-prefetchable) [size=16K]
    Capabilities: <access denied>
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel

07:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1aec] (rev a1) (prog-if 30 [XHCI])
    Subsystem: NVIDIA Corporation Device [10de:1324]
    Flags: bus master, fast devsel, latency 0, IRQ 49
    Memory at e2000000 (64-bit, prefetchable) [size=256K]
    Memory at e2040000 (64-bit, prefetchable) [size=64K]
    Capabilities: <access denied>
    Kernel driver in use: xhci_hcd

07:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1aed] (rev a1)
    Subsystem: NVIDIA Corporation Device [10de:1324]
    Flags: bus master, fast devsel, latency 0, IRQ 11
    Memory at f7084000 (32-bit, non-prefetchable) [size=4K]
    Capabilities: <access denied>
    Kernel driver in use: vfio-pci

These are in the same iommu group (15). You can see that some devices of the group are not being held by vfio-pci. Since I have a NVIDIA card in the guest and host, I cannot simply blacklist the nvidia driver, and I didnt bother to blacklist the others. Instead I did this script to manually unbind pci devices from drivers:

echo -n "0000:07:00.1" > /sys/bus/pci/drivers/snd_hda_intel/unbind
echo -n "0000:07:00.1" > /sys/bus/pci/drivers/vfio-pci/bind

echo -n "0000:07:00.2" > /sys/bus/pci/drivers/xhci_hcd/unbind
echo -n "0000:07:00.2" > /sys/bus/pci/drivers/vfio-pci/bind

echo -n "0000:07:00.3" > /sys/bus/pci/drivers/nvidia-gpu/unbind
echo -n "0000:07:00.3" > /sys/bus/pci/drivers/vfio-pci/bind

Sometimes the 07.00.3 lines aren't needed because the driver starts bound to vfio-pci but sometimes its being used by nvidia-gpu.

After I run the script as root, all the 4 devices show up as being used by vfio-pci.

If I add only the 07.00.0, which is the GTX 1060, virt-manager starts instantly and Windows shows the GTX 1060 in the device manager, but with an yellow exclamation mark. I tried updating the drivers through device manager but it does nothing. If I start a game on Steam, it starts and closes. I obviously change the HDMI of the TV to the one of GTX 1060 but I get "no signal" from the TV. Windows also installed NVIDIA control panel but when I try to launch it, nothing happens. It's worth noticing that when I start virt-manager with GTX, I cannot detect other displays, only the default one, and I can't even change the resolution (800x600). I tried QXL, VGA and Virtio, all of them start at 800x600 without letting me change.

If I put the GTX (07.00.0) with any other of the 07.00.x for x=1,2,3, virt-manager hangs for some minutes when I press run, until it finally starts the VM. Then Windows starts extremelly slow, I can't even unlock the screen, because when I click in it it takes like 5 minutes to animate and show the password input for me to unlock it.

My Windows 10 is using i440FX as chipset and UEFI x86_64: /usr/share/OVMF/OVMF_CODE.fd as firmware.

Im using Ubuntu 19.10 with kernel 4.15.0-1050-oem because the one that comes (5.something) gives this freeze error: 19.10 Ubuntu host entirely freezes during Windows 10 qemu install everytime I launch the windows vm.

I then compiled qemu 4.1, but when I launch with all the 4 pci devices (07.00.x for x=0,1,2,3), I stil get the problem of ultra slow virt-manager start and also ultra slow Windows unlock screen. When I launch with just the GTX (07.00.0), I get the same behavior as with the old QEMU again.

It's worth noticing that everytime I add the 07.00.x for x=1,2,3, when I turn off the VM and try to remove these pci devices, virt-manager freezes and I have to kill it and reopen it. Then when it opens I have to wait for QEMU/KVM to connect again. When it does, I get

Error starting domain: internal error: Unknown PCI header type '127'

when I start the VM again. I have to reboot the host. This happens with the old and the 4.1 qemu.

Here's my detailed XML machine: https://pastebin.com/BKtJq9PX

I tried changing the 

<type arch="x86_64" machine="pc-i440fx-4.0">hvm</type>

to

<type arch='x86_64' machine='pc-q35-4.1'>hvm</type>

but I always get

error: XML error: The PCI controller with index='0' must be model='pcie-root' for this machine type, but model='pci-root' was found instead

Any ideas on how to further debug this problem?

UPDATE:

I found out that the error in my NVIDIA driver was "error 43", and thus found about https://mathiashueber.com/fighting-error-43-nvidia-gpu-virtual-machine/. I applied the patches in the XML but they didn't work.

Here's my entire XML: https://pastebin.com/yajTNFPs

Improve this question
1

You need to make sure the devices you are passing through aren't tainted by the drivers on the host side before they are assigned to vfio-pci. Typically you would do this by blacklisting nouveau, nvidia and any other drivers invoiced (e.g. drivers for the USB devices mentioned above). If you need those drivers for other devices in the system, load them in a startup script AFTER the devices that need to be bound to vfio-pci are bound to vfio-pci.

Also be aware that binding and unbinding PCI devices between guest and host dynamically causes stability issues on many systems. The best way is to statically bind the devices to vfio-pci and leave them bound.

Your libvirt xml file looks like it has all the necessary bits to avoid the code 43 error in the guest.

Improve this answer
  • one of the problems was exactly what you mentioned: the drivers tainted the GPU. I simply switched the slot in which the OS boot so the GPU for the VM was left untouched and it worked. However it only works if I pass the GPU VGA device only. If I try to pass it with any combination of [audio pci device, usb controller pci device, serial pci device] of the GPU, I get the problem of the machine booting in an extremely slow way such that it takes 10 minutes to get to the login screen. For that problem I didn't find any solutions, but I did find someone who also had it. Do you have any idea? – Guerlando OCs May 6 '20 at 4:47
  • Did you make sure those devices are also untainted by the host drivers? My approach is to blacklist the host side drivers that those devices use, then load them manually if needed for other host devices after explicitly binding the VM devices to vfio-pci in a startup script. – Gordan Bobic May 6 '20 at 9:08
  • I made 3 of 4 devices start bound to vfio-pci. I think the USB starts bound to xhci_hcd and I couldn't fix that. However it's not the USB one that I pass to PCI, I tried passing only VGA and Audio, which do start bound to vfio-pci. Take a look here: askubuntu.com/questions/1212969/softdep-for-vfio-pci-wont-work it shows the 4 devices – Guerlando OCs May 6 '20 at 20:06
  • You have to pass all of the devices in the same IOMMU group together. I have never seen all functions of a device not being part of the same IOMMU group, so you'll have to pre-emptively bind all of those device to vfio-pci before any other driver claims them. The way I do it is to blacklist all drivers that naturally bind to the devices I want to bind to vfio-pci, create a startup script (I use /etc/syscnfig/modules/ hooks on CentOS) that does the binding, and then after the devices I am passing through are all bound, it explicitly invokes modprobe on the drivers I blacklisted. – Gordan Bobic May 6 '20 at 20:31
  • That lets me pass one Nvidia GPU and USB controller to the VM while having an identical GPU and USB controller for the host. – Gordan Bobic May 6 '20 at 20:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.