libvirt_win10_vm
libvirt_win10_vm copied to clipboard
tuh8888
- Virtual Machine with vGPU Unlock for single GPU desktop This document catalogs my working setup for configuring an Arch Linux host OS to run a Windows 10 VM while passing through a virtualized GPU.
** Sources The resources I used during my own install. This document is a compilation of the instructions and files provided by these people.
*** Guides - [[https://docs.google.com/document/d/1pzrWJ9h-zANCtyqRgS7Vzla0Y8Ea2-5z2HEi4X75d2Q][vGPU Wiki]] - [[https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF][PCI passthrough via OVMF]] - [[https://wvthoog.nl/proxmox-7-vgpu/][Proxmox 7 vGPU]] by wvthoog - [[https://www.youtube.com/watch?v=cPrOoeMxzu0][vGPU YouTube guide]] by Craft Computing - [[https://looking-glass.io/docs/stable/install/][Looking Glass]] - [[https://www.youtube.com/watch?v=wEhvQEyiOwI][Looking Glass YouTube guide]] by Pavol Elsig
*** Sources - [[https://github.com/DualCoder/vgpu_unlock][vGPU unlock scripts]] by DualCoder - [[https://github.com/KrutavShah/vGPU_LicenseBypass][vGPU_LicenseBypass]] by KrutavShah - [[https://github.com/rupansh/vgpu_unlock_5.12][vGPU unlock patch for kernel 5.12]] by rupansh
** Configuration The what and how. Mostly described in suggested installation order unless otherwise stated. *** Host **** Operating system Configuration of the host OS for virtualization using KVM and device passthrough using OVMF, IOMMU, and VFIO.
***** Arch Linux
***** Kernel 5.12.15-arch1-1
***** Kernel modules - vfio - vfio_iommu_type1 - vfio_pci - vfio_virqfd ****** How to add 1. Edit [[file:/etc/mkinitcpio.conf]] 2. Add ~MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...)~ to the MODULES line. 3. Add ~HOOKS=(... modconf ...)~ to the HOOKS line. 4. Update ~update-initramfs -P~ (use ~mkinitpico -P~ for manjaro)
***** Boot parameters - amd_iommu=on - iommu=pt - vfio_iommu_type1.allow_unsafe_interrupts=1 - kvm.ignore_msrs=1 - blacklist=nouveau ****** How to add 1. edit [[file:/etc/default/grub]] 2. Add to GRUB_CMDLINE_LINUX_DEFAULT: #+begin_src GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 blacklist=nouveau ..." #+end_src 3. Update GRUB ~update-grub~
**** Hardware The physical devices present on my system.
- CPU: AMD Ryzen 7 5800X (16) @ 3.800GHz
- GPU: NVIDIA GeForce GTX 1060 6GB
- Memory: 16MiB
- 1440p Monitor
**** Software What needs to be installed.
***** Arch Packages :PROPERTIES: :ID: 94df3230-029b-42c2-a7bc-db75af7fb82a :END: Packages installed using =pacman=.
****** linux
The OS kernel.
- Version: 5.12.15
******* How to install: First option 1. Install ~downgrade~ from AUR (using ~yay~ or other AUR helper) 2. ~downgrade linux~ 3. Select 5.12.15
******* How to install: From Arch archive 1. Download package ~curl -O https://archive.archlinux.org/packages/l/linux/linux-5.12.15.arch1-1-x86_64.pkg.tar.zst~ 2. Install with ~pacman -U linux-5.12.15.arch1-1-x86_64.pkg.tar.zst~
****** linux-headers
Needed to build kernel modules, particularly the Nvidia driver.
- Version: 5.12.15
******* How to install: First option 1. Install ~downgrade~ from AUR (using ~yay~ or other AUR helper) 2. ~downgrade linux-headers~ 3. Select 5.12.15
******** From Arch archive 1. Download package ~curl -O https://archive.archlinux.org/packages/l/linux-headers/linux-headers-5.12.15.arch1-1-x86_64.pkg.tar.zst~ 2. Install with ~pacman -U linux-headers-5.12.15.arch1-1-x86_64.pkg.tar.zst~
****** virt-manager
Used to create and run virtual machines.
****** libvirt
****** kvm
(Kernel-based Virtual Machine). A hypervisor. Used for virtualization.
****** qemu
****** looking-glass
Used to view VM in graphical display.
****** iptables-nft
For networking with the VM.
****** edk2-ovmf
****** dnsmasq
Used for VM networking.
****** mdevctl
Used to create vGPU devices
****** scream
Used to read audio stream from network.
******* How to setup 1. Install using package manager. 2. Create [[https://github.com/duncanthrax/scream/issues/155#issuecomment-924401541][systemd-file]] #+BEGIN_SRC systemd [Unit] Description=Scream network audio client
[Unit] Description=Scream Receiver After=pulseaudio.service network-online.target Wants=pulseaudio.service
[Service] Type=simple ExecStartPre=/bin/sleep 3 ExecStart=scream -i virbr0 Restart=always RestartSec=1
[Install] WantedBy=default.target #+END_SRC
***** Python packages :PROPERTIES: :ID: a7fa4275-7f2e-457b-ab7b-00af5a52de36 :END: Installed with =pip=. ****** frida ******* How to install ******** If using conda 1. Install ~python-pip~ using pacman. 2. Run ~/bin/pip install frida~ ******** Otherwise 1. ~pip install frida~
***** Graphics setup
****** Nvidia Driver ******* Version: 460.73.01 merged and pre-unlocked This driver is a merged version of Nvidia's vGPU driver and their standard driver. It allows for the host graphics to be displayed while the vGPU's are active. ******* Components :PROPERTIES: :ID: 52f0c050-0f82-48af-bd19-bd5c613a29b3 :END: 1. [[https://drive.google.com/file/d/1dCyUteA2MqJaemRKqqTu5oed5mINu9Bw/view?usp=sharing][merged driver]] 2. [[https://github.com/DualCoder/vgpu_unlock][vgpu_unlock]] 3. [[https://raw.githubusercontent.com/rupansh/vgpu_unlock_5.12/master/twelve.patch][vgpu_unlock_5.12 patch]]
******* How to install ******** Make sure all above [[id:94df3230-029b-42c2-a7bc-db75af7fb82a][arch packages]] and [[id:a7fa4275-7f2e-457b-ab7b-00af5a52de36][python packages]] are installed. ******** Download the[[id:52f0c050-0f82-48af-bd19-bd5c613a29b3][components]] using ~curl -O the_link~ ******** Make the driver executable. ~chmod +x NVIDIA-Linux-x86_64-460.73.01-grid-vgpu-kvm-v5.run~ ******** Extract it. ~./NVIDIA-Linux-x86_64-460.73.01-grid-vgpu-kvm-v5.run -x~ ******** ~cd NVIDIA-Linux-x86_64-640.73.01-grid-vgpu-kvm-v5~ ******** Apply the 5.12 patch ~patch -p0 < ../twelve.patch~ ******** Disable graphics ~sudo systemctl isolate multi-user.target~ ******** Re-login as root and navigate back to the driver directory #+BEGIN_SRC bash cd /home/user/Downloads/NVIDIA-Linux-x86_64-460.73.01-grid-vgpu-kvm-v5 #+END_SRC ******** Install the driver #+BEGIN_SRC bash ./nvidia-installer --dkms #+END_SRC ******** Add ~vgpu_unlock~ to service files. #+BEGIN_SRC bash nano /lib/systemd/system/nvidia-vgpud.service #+end_src
Replace ExecStart with:
#+BEGIN_SRC systemd
ExecStart=/root/vgpu_unlock/vgpu_unlock /usr/bin/nvidia-vgpud #+END_SRC
#+begin_src bash
nano /lib/systemd/system/nvidia-vgpu-mgr.service #+end_src
Replace ExecStart with:
#+BEGIN_SRC systemd
ExecStart=/root/vgpu_unlock/vgpu_unlock /usr/bin/nvidia-vgpu-mgr #+END_SRC
#+begin_src bash
systemctl daemon-reload #+end_src ******** Add ~vgpu_unlock~ to source files. #+BEGIN_SRC bash nano /usr/src/nvidia-460.73.01/nvidia/os-interface.c #+END_SRC
Under #include “nv-time.h” insert this line
#+BEGIN_SRC cpp
#include "/root/vgpu_unlock/vgpu_unlock_hooks.c" #+END_SRC #+BEGIN_SRC bash nano /usr/src/nvidia-460.73.01/nvidia/nvidia.Kbuild #+END_SRC
Add this to the bottom of the file.
#+BEGIN_SRC
ldflags-y += -T /root/vgpu_unlock/kern.ld #+END_SRC ******** Remove and reinstall the Nvidia dkms module #+BEGIN_SRC bash dkms remove -m nvidia -v 460.73.01 --all dkms install -m nvidia -v 460.73.01 #+END_SRC
******** Make all of vgpu_unlock executable
~chmod -R +x vgpu_unlock~
******** Reboot
****** vGPU Profiles - nvidia-48 GRID P40-3Q
******* How to install
******** Choose profile :PROPERTIES: :ID: 492ad298-5b52-4cb4-913c-a444940d9492 :END: #+BEGIN_SRC bash :results verbatim mdevctl types #+END_SRC
#+RESULTS:
#+begin_example
0000:2b:00.0
nvidia-156
Available instances: 0
Device API: vfio-pci
Name: GRID P40-2B
Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
nvidia-215
Available instances: 0
Device API: vfio-pci
Name: GRID P40-2B4
Description: num_heads=4, frl_config=45, framebuffer=2048M, max_resolution=5120x2880, max_instance=12
nvidia-241
Available instances: 0
Device API: vfio-pci
Name: GRID P40-1B4
Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
nvidia-283
Available instances: 0
Device API: vfio-pci
Name: GRID P40-4C
Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=4096x2160, max_instance=6
nvidia-284
Available instances: 0
Device API: vfio-pci
Name: GRID P40-6C
Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=4096x2160, max_instance=4
nvidia-285
Available instances: 0
Device API: vfio-pci
Name: GRID P40-8C
Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=4096x2160, max_instance=3
nvidia-286
Available instances: 0
Device API: vfio-pci
Name: GRID P40-12C
Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=4096x2160, max_instance=2
nvidia-287
Available instances: 0
Device API: vfio-pci
Name: GRID P40-24C
Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=4096x2160, max_instance=1
nvidia-46
Available instances: 0
Device API: vfio-pci
Name: GRID P40-1Q
Description: num_heads=4, frl_config=60, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
nvidia-47
Available instances: 0
Device API: vfio-pci
Name: GRID P40-2Q
Description: num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=7680x4320, max_instance=12
nvidia-48
Available instances: 6
Device API: vfio-pci
Name: GRID P40-3Q
Description: num_heads=4, frl_config=60, framebuffer=3072M, max_resolution=7680x4320, max_instance=8
nvidia-49
Available instances: 0
Device API: vfio-pci
Name: GRID P40-4Q
Description: num_heads=4, frl_config=60, framebuffer=4096M, max_resolution=7680x4320, max_instance=6
nvidia-50
Available instances: 0
Device API: vfio-pci
Name: GRID P40-6Q
Description: num_heads=4, frl_config=60, framebuffer=6144M, max_resolution=7680x4320, max_instance=4
nvidia-51
Available instances: 0
Device API: vfio-pci
Name: GRID P40-8Q
Description: num_heads=4, frl_config=60, framebuffer=8192M, max_resolution=7680x4320, max_instance=3
nvidia-52
Available instances: 0
Device API: vfio-pci
Name: GRID P40-12Q
Description: num_heads=4, frl_config=60, framebuffer=12288M, max_resolution=7680x4320, max_instance=2
nvidia-53
Available instances: 0
Device API: vfio-pci
Name: GRID P40-24Q
Description: num_heads=4, frl_config=60, framebuffer=24576M, max_resolution=7680x4320, max_instance=1
nvidia-54
Available instances: 0
Device API: vfio-pci
Name: GRID P40-1A
Description: num_heads=1, frl_config=60, framebuffer=1024M, max_resolution=1280x1024, max_instance=24
nvidia-55
Available instances: 0
Device API: vfio-pci
Name: GRID P40-2A
Description: num_heads=1, frl_config=60, framebuffer=2048M, max_resolution=1280x1024, max_instance=12
nvidia-56
Available instances: 0
Device API: vfio-pci
Name: GRID P40-3A
Description: num_heads=1, frl_config=60, framebuffer=3072M, max_resolution=1280x1024, max_instance=8
nvidia-57
Available instances: 0
Device API: vfio-pci
Name: GRID P40-4A
Description: num_heads=1, frl_config=60, framebuffer=4096M, max_resolution=1280x1024, max_instance=6
nvidia-58
Available instances: 0
Device API: vfio-pci
Name: GRID P40-6A
Description: num_heads=1, frl_config=60, framebuffer=6144M, max_resolution=1280x1024, max_instance=4
nvidia-59
Available instances: 0
Device API: vfio-pci
Name: GRID P40-8A
Description: num_heads=1, frl_config=60, framebuffer=8192M, max_resolution=1280x1024, max_instance=3
nvidia-60
Available instances: 0
Device API: vfio-pci
Name: GRID P40-12A
Description: num_heads=1, frl_config=60, framebuffer=12288M, max_resolution=1280x1024, max_instance=2
nvidia-61
Available instances: 0
Device API: vfio-pci
Name: GRID P40-24A
Description: num_heads=1, frl_config=60, framebuffer=24576M, max_resolution=1280x1024, max_instance=1
nvidia-62
Available instances: 0
Device API: vfio-pci
Name: GRID P40-1B
Description: num_heads=4, frl_config=45, framebuffer=1024M, max_resolution=5120x2880, max_instance=24
#+end_example
I chose nvidia-47 (GRID P40-3Q) because it has 3Q at the end of its name. This means I can have two 3GB vGPU's running.
******** Determine GPU PCI address #+BEGIN_SRC bash :results verbatim lspci | grep VGA PCI_ADDRESS=$(lspci | grep VGA | grep -Po '\d\w:\d\d.\d') echo $PCI_ADDRESS #+END_SRC
#+RESULTS:
: 2b:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] (rev a1)
: 2b:00.0
******** Generate UUIDs for each vGPU #+BEGIN_SRC bash UUID1=$(uuidgen) UUID2=$(uuidgen) echo $UUID1 $UUID2 #+END_SRC
#+RESULTS:
: 2e82a1ad-3c7a-4b79-869f-3637b6b299a7 7a9c2eda-fa37-49f7-bb97-069c679b6eef
******** Create vGPUs :PROPERTIES: :ID: 62088cb8-6c34-4fa0-9678-23bdc78632b6 :END: #+BEGIN_SRC bash mdevctl start -u $UUID1 -p 0000:$PCI_ADDRESS -t nvidia-48 mdevctl start -u $UUID2 -p 0000:$PCI_ADDRESS -t nvidia-48
mdevctl define -a -u $UUID1 mdevctl define -a -u $UUID2 #+END_SRC
**** VM Image The virtual machine configuration.
- Chipset: Q35
- Firmware: UEFI x86_64: /usr/share/edk2-ovmf/x64/OVMF_CODE.fd
- Hypervisor KVM
- Spice Server
- Video QXL
- MDEV
- shmem looking-glass
***** How to install
****** In virt-manager, create virtual machine.
****** Apply above settings minus "MDEV" and "shmem looking-glass".
****** Start VM and install OS.
****** Shutdown guest.
****** Add looking-glass shmem device.
The size is determined like [[https://looking-glass.io/docs/stable/install/#client-determining-memory][this]].
#+BEGIN_SRC xml
Make a file =/etc/tmpfiles.d/10-looking-glass.conf=
#+BEGIN_SRC conf
#Type Path Mode UID GID Age Argument
f /dev/shm/looking-glass 0660 user kvm - #+END_SRC ****** Remove tablet device
****** Add MDEV device.
Where ~$UUID1~ is the UUID of one of the mdev vGPU's created [[id:62088cb8-6c34-4fa0-9678-23bdc78632b6][earlier]].
#+BEGIN_SRC xml
***** How to run
1. Start VM. (May need to click run several times if error related to "vfio-
*** Guest Inside the VM.
**** Operating system - Windows 10
**** To download (within guest) :PROPERTIES: :ID: 6d9a90c9-e736-4d73-8447-a4e7db5c5bde :END: - [[https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.208-1/virtio-win-guest-tools.exe][virtio guest tools]] - [[https://www.spice-space.org/download/windows/spice-guest-tools/spice-guest-tools-latest.exe][spice guest tools]] - [[https://looking-glass.io/ci/host/download?id=stable][looking glass host]] - [[https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/upstream-virtio/virtio-win10-prewhql-0.1-161.zip][virtio PCI RAM controller driver]] - [[https://storage.googleapis.com/nvidia-drivers-us-public/GRID/GRID12.2/462.31_grid_win10_server2016_server2019_64bit_international.exe][Nvidia GRID driver]] - [[https://github.com/KrutavShah/vGPU_LicenseBypass/releases/download/v1.0/vGPULicenseBypass.bat][vGPU_LicenseBypass]] - [[https://github.com/duncanthrax/scream][Scream audio driver]]
**** Software :PROPERTIES: :ID: ae17e44e-f00c-4995-a017-8dcd919c8f0f :END: ***** Looking Glass Host ****** How to install 1. Download [[id:6d9a90c9-e736-4d73-8447-a4e7db5c5bde][components]]. 2. Update driver in =Device Manager->System Devicess->PCI standard RAM Controller= to [[https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/upstream-virtio/virtio-win10-prewhql-0.1-161.zip][virtio PCI RAM controller driver]]. 3. Install [[https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.208-1/virtio-win-guest-tools.exe][virtio guest tools]] and [[https://www.spice-space.org/download/windows/spice-guest-tools/spice-guest-tools-latest.exe][spice guest tools]]. 4. Install [[https://looking-glass.io/ci/host/download?id=stable][looking glass host]] by running =looking-glass-host-setup.exe= as administrator. 5. VM.
***** [[https://storage.googleapis.com/nvidia-drivers-us-public/GRID/GRID12.2/462.31_grid_win10_server2016_server2019_64bit_international.exe][Nvidia GRID driver]] :PROPERTIES: :ID: e152ccec-fd25-4c23-befd-feb84becdf70 :END: ****** How to install 1. Run [[https://storage.googleapis.com/nvidia-drivers-us-public/GRID/GRID12.2/462.31_grid_win10_server2016_server2019_64bit_international.exe][Nvidia GRID driver]] installer. 2. Reboot. 3. In =Nvidia Console= set resolution. My monitor used 2560x1440. 4. Run [[https://github.com/KrutavShah/vGPU_LicenseBypass/releases/download/v1.0/vGPULicenseBypass.bat][vGPU_LicenseBypass]].
# TODO Maybe add Windows Autologon instructions https://docs.microsoft.com/en-us/sysinternals/downloads/autologon
# TODO Maybe add looking-glass-host start-up instructions
***** [[https://github.com/duncanthrax/scream/releases/download/3.8/Scream3.8.zip][Scream audio driver]]
****** How to install 1. Install driver. 2. Change sound source to Scream (WDM).
** My files
- [[file:win10.xml][Final QEMU Windows 10 VM image]]
- [[file:mdev.sh][mdev device creation script]]
** Suggestions Since there is a lot of messing with the kernel and graphics drivers, it's important to have a plan for recovering your host OS.
- Use timeshift or other auto-snapshotting of the host.
- Have a recovery drive available.
** Maintenance
*** Preventing accidental upgrades In order to avoid accidentally upgrading these carefully chosen package versions, make sure to ignore updates to them:
~IgnorePkg = linux linux-headers nvidia-dkms nvidia nvidia-utils nvidia-settings opencl-nvidia libxnvctrl~
tuh8888