I'm trying to do GPU passthrough for GTX 1060 on Ryzen 7 2700x.
I followed https://mathiashueber.com/windows-virtual-machine-gpu-passthrough-ubuntu/ but I couldn't blacklist the nvidia driver as the host's GPU is also NVIDIA, so I use quirks mentioned below.
When I start Ubuntu 19 in the host machine, I get this in my
07:00.0 VGA compatible controller : NVIDIA Corporation TU116 [GeForce GTX 1660] [10de:2184] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation TU116 [GeForce GTX 1660] [10de:1324] Flags: bus master, fast devsel, latency 0, IRQ 11 Memory at f6000000 (32-bit, non-prefetchable) [size=16M] Memory at d0000000 (64-bit, prefetchable) [size=256M] Memory at e0000000 (64-bit, prefetchable) [size=32M] I/O ports at f000 [size=128] Expansion ROM at 000c0000 [disabled] [size=128K] Capabilities: <access denied> Kernel driver in use: vfio-pci Kernel modules: nvidiafb, nouveau 07:00.1 Audio device : NVIDIA Corporation Device [10de:1aeb] (rev a1) Subsystem: NVIDIA Corporation Device [10de:1324] Flags: bus master, fast devsel, latency 0, IRQ 81 Memory at f7080000 (32-bit, non-prefetchable) [size=16K] Capabilities: <access denied> Kernel driver in use: snd_hda_intel Kernel modules: snd_hda_intel 07:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1aec] (rev a1) (prog-if 30 [XHCI]) Subsystem: NVIDIA Corporation Device [10de:1324] Flags: bus master, fast devsel, latency 0, IRQ 49 Memory at e2000000 (64-bit, prefetchable) [size=256K] Memory at e2040000 (64-bit, prefetchable) [size=64K] Capabilities: <access denied> Kernel driver in use: xhci_hcd 07:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1aed] (rev a1) Subsystem: NVIDIA Corporation Device [10de:1324] Flags: bus master, fast devsel, latency 0, IRQ 11 Memory at f7084000 (32-bit, non-prefetchable) [size=4K] Capabilities: <access denied> Kernel driver in use: vfio-pci
These are in the same iommu group (15). You can see that some devices of the group are not being held by vfio-pci. Since I have a NVIDIA card in the guest and host, I cannot simply blacklist the nvidia driver, and I didnt bother to blacklist the others. Instead I did this script to manually unbind pci devices from drivers:
echo -n "0000:07:00.1" > /sys/bus/pci/drivers/snd_hda_intel/unbind echo -n "0000:07:00.1" > /sys/bus/pci/drivers/vfio-pci/bind echo -n "0000:07:00.2" > /sys/bus/pci/drivers/xhci_hcd/unbind echo -n "0000:07:00.2" > /sys/bus/pci/drivers/vfio-pci/bind echo -n "0000:07:00.3" > /sys/bus/pci/drivers/nvidia-gpu/unbind echo -n "0000:07:00.3" > /sys/bus/pci/drivers/vfio-pci/bind
07.00.3 lines aren't needed because the driver starts bound to
vfio-pci but sometimes its being used by
After I run the script as root, all the 4 devices show up as being used by
If I add only the 07.00.0, which is the GTX 1060, virt-manager starts instantly and Windows shows the GTX 1060 in the device manager, but with an yellow exclamation mark. I tried updating the drivers through device manager but it does nothing. If I start a game on Steam, it starts and closes. I obviously change the HDMI of the TV to the one of GTX 1060 but I get "no signal" from the TV. Windows also installed NVIDIA control panel but when I try to launch it, nothing happens. It's worth noticing that when I start virt-manager with GTX, I cannot detect other displays, only the default one, and I can't even change the resolution (800x600). I tried QXL, VGA and Virtio, all of them start at 800x600 without letting me change.
If I put the GTX (07.00.0) with any other of the
07.00.x for x=1,2,3, virt-manager hangs for some minutes when I press run, until it finally starts the VM. Then Windows starts extremelly slow, I can't even unlock the screen, because when I click in it it takes like 5 minutes to animate and show the password input for me to unlock it.
My Windows 10 is using
i440FX as chipset and
UEFI x86_64: /usr/share/OVMF/OVMF_CODE.fd as firmware.
Im using Ubuntu 19.10 with kernel 4.15.0-1050-oem because the one that comes (5.something) gives this freeze error: 19.10 Ubuntu host entirely freezes during Windows 10 qemu install everytime I launch the windows vm.
I then compiled qemu 4.1, but when I launch with all the 4 pci devices (
07.00.x for x=0,1,2,3), I stil get the problem of ultra slow virt-manager start and also ultra slow Windows unlock screen. When I launch with just the GTX (07.00.0), I get the same behavior as with the old QEMU again.
It's worth noticing that everytime I add the
07.00.x for x=1,2,3, when I turn off the VM and try to remove these pci devices, virt-manager freezes and I have to kill it and reopen it. Then when it opens I have to wait for QEMU/KVM to connect again. When it does, I get
Error starting domain: internal error: Unknown PCI header type '127'
when I start the VM again. I have to reboot the host. This happens with the old and the 4.1 qemu.
Here's my detailed XML machine: https://pastebin.com/BKtJq9PX
I tried changing the
<type arch="x86_64" machine="pc-i440fx-4.0">hvm</type>
<type arch='x86_64' machine='pc-q35-4.1'>hvm</type>
but I always get
error: XML error: The PCI controller with index='0' must be model='pcie-root' for this machine type, but model='pci-root' was found instead
Any ideas on how to further debug this problem?
I found out that the error in my NVIDIA driver was "error 43", and thus found about https://mathiashueber.com/fighting-error-43-nvidia-gpu-virtual-machine/. I applied the patches in the XML but they didn't work.
Here's my entire XML: https://pastebin.com/yajTNFPs