[viogpu3d] Virtio GPU 3D acceleration for windows

[max8rr8]

Hello! This series of changes spanning across multiple repositories introduce support for 3d accelerated virtiogpu windows guests.

Wglgears window is rendered with wgl on virgl and window below it is cube rendered with d3d10umd on virgl.

How to test

NOTE: This driver does have some rendering glitches and might crash. Try at your own risk. 0. Create qemu windows VM with VirtIO GPU with 3d acceleration enabled. It is highly recommended to use "disposable" virtual machine to test, loss of data might occur.

1. Use patched version of virglrenderer from this repo branch viogpu_win
2. Compile from source OR download pre-built drivers.
3. Install drivers on target VM. Note: if drivers were not signed you need to manually select them in device manager.

Known issues

• FIXED: ~~Frames displayed on screen are lagging behind~~

• FIXED: ~~D3d10 clearing color is not supported~~

• FIXED: ~~D3d10 applications using DXGI_SWAP_EFFECT_DISCARD and DXGI_SWAP_EFFECT_SEQUENTIAL are not displayed.~~

• Rendering glitches in WinUI3 apps.

  There are some rendering glitches in apps based on WinUI3 (maybe other apps too), best way to see them is to install WinUI3 Gallery from microsoft store and navigate around it. Haven't yet invistigated.

• Vscode (possibly other electron apps) does not render

  Black window. Requires implementation of PIPE_QUERY_TIMESTAMP_DISJOINT in virglrenderer.

• No preemption

  Kernel-mode driver does not implement preemption, and i am very confised about how to implement it in WDDM. VioGpu3D disables preemption systemwide to workaround lack of preemption implementation, but this is not ideal. Would appreciate some help.

Siblings

• Mesa: https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24223
• kvm-guest-drivers: https://github.com/virtio-win/kvm-guest-drivers-windows/pull/943
• virglrenderer: https://gitlab.freedesktop.org/virgl/virglrenderer/-/merge_requests/1185

[vrozenfe]

@max8rr8 Hi Max,

Thank for a very impressive work you've done. Please give us some time to go through your code and see how to integrate it into our upstream repository.

Nice work! All the best, Vadim.

[max8rr8]

Changes:

• Fixed crash caused by use of PAGED_CODE at DISPATCH_LEVEL in flip timer. Instead of using ExTimer, separate flipping thread is used.
• Building system now checks for viogpu related dll's specifically when deciding whether to compile viogpu3d
• Renamed DLL's and updated build instructions due to changes requested by mesa

[max8rr8]

Changes:

• Re-implemented RotateResourceIdentites. This fixed biggest issue of frames lagging behind, now using driver becomes pretty smooth experience.
• Implemented clear_render_target similarly to i915
• Implemented support for staging resources and kernel-mode Present BLT which fixes window display in applications using DXGI_SWAP_EFFECT_DISCARD and DXGI_SWAP_EFFECT_DISCARD.

Pre-built driver provided in description was updated with these changes.

[mincore]

I've tested the driver on win10, and got a black screen. But if change the guest os to ubuntu(same qemu command line), glxgears performs well. (glxinfo shows that the renderer is NVIDIA gpu)

env

guest os: win 10 enterprise ltsc + viogpu3d

host os: fedora38, qemu 8.0.3 with virglrenderer

qemu command line

-display egl-headless,rendernode=/dev/dri/card1 -device virtio-vga-gl -trace enable="virtio_gpu*" -D qemu.log

virglrenderer.log

gl_version 46 - core profile enabled
GLSL feature level 460
vrend_check_no_error: context error reported 3 "" Unknown 1282
context 3 failed to dispatch DRAW_VBO: 22
vrend_decode_ctx_submit_cmd: context error reported 3 "" Illegal command buffer 786440
GLSL feature level 460
vrend_check_no_error: context error reported 5 "" Unknown 1282
context 5 failed to dispatch DRAW_VBO: 22
vrend_decode_ctx_submit_cmd: context error reported 5 "" Illegal command buffer 786440
GLSL feature level 460
GLSL feature level 460
GLSL feature level 460
GLSL feature level 460
GLSL feature level 460
vrend_check_no_error: context error reported 9 "" Unknown 1282
context 9 failed to dispatch DRAW_VBO: 22

qemu_strace.log

virtio_gpu_features virgl 0
virtio_gpu_features virgl 0
virtio_gpu_cmd_get_edid scanout 0
virtio_gpu_cmd_get_display_info
virtio_gpu_cmd_ctx_create ctx 0x1, name
virtio_gpu_cmd_ctx_create ctx 0x2, name
virtio_gpu_cmd_res_create_3d res 0x1, fmt 0x1, w 1280, h 1024, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x2, res 0x1
virtio_gpu_cmd_res_back_attach res 0x1
virtio_gpu_cmd_res_create_3d res 0x2, fmt 0x1, w 1280, h 1024, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x2, res 0x2
virtio_gpu_cmd_res_create_2d res 0x3, fmt 0x2, w 1280, h 1024
virtio_gpu_cmd_res_back_attach res 0x3
virtio_gpu_cmd_set_scanout id 0, res 0x1, w 1280, h 1024, x 0, y 0
virtio_gpu_cmd_res_flush res 0x1, w 1280, h 1024, x 0, y 0
virtio_gpu_cmd_set_scanout id 0, res 0x1, w 1280, h 1024, x 0, y 0
virtio_gpu_cmd_res_flush res 0x1, w 1280, h 1024, x 0, y 0
virtio_gpu_cmd_ctx_create ctx 0x3, name
virtio_gpu_cmd_ctx_destroy ctx 0x3
virtio_gpu_cmd_ctx_create ctx 0x3, name
virtio_gpu_cmd_res_create_3d res 0x4, fmt 0xb1, w 48, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x4
virtio_gpu_cmd_res_create_3d res 0x5, fmt 0xb1, w 4000, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x5
virtio_gpu_cmd_res_create_3d res 0x6, fmt 0xb1, w 16, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x6
virtio_gpu_cmd_res_create_3d res 0x7, fmt 0xb1, w 48, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x7
virtio_gpu_cmd_res_create_3d res 0x8, fmt 0xb1, w 240012, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x8
virtio_gpu_cmd_res_create_3d res 0x9, fmt 0xb1, w 102400, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x9
virtio_gpu_cmd_res_create_3d res 0xa, fmt 0xb1, w 144, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0xa
virtio_gpu_cmd_res_create_3d res 0xb, fmt 0xb1, w 160000, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0xb
virtio_gpu_cmd_res_create_3d res 0xc, fmt 0xb1, w 16000, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0xc
virtio_gpu_cmd_res_create_3d res 0xd, fmt 0xb1, w 240000, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0xd
virtio_gpu_cmd_res_create_3d res 0xe, fmt 0xb1, w 192, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0xe
virtio_gpu_cmd_res_create_3d res 0xf, fmt 0xb1, w 16, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0xf
virtio_gpu_cmd_res_create_3d res 0x10, fmt 0xb1, w 272, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x10
virtio_gpu_cmd_res_create_3d res 0x11, fmt 0xb1, w 240, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x11
virtio_gpu_cmd_res_create_3d res 0x12, fmt 0xb1, w 272, h 1, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x12
virtio_gpu_cmd_res_create_3d res 0x13, fmt 0x1, w 50, h 50, d 1
virtio_gpu_cmd_ctx_res_attach ctx 0x3, res 0x13
virtio_gpu_cmd_ctx_create ctx 0x4, name
virtio_gpu_cmd_res_back_attach res 0x13
virtio_gpu_cmd_res_back_attach res 0x4
virtio_gpu_cmd_res_back_attach res 0x5
virtio_gpu_cmd_res_back_attach res 0x6
virtio_gpu_cmd_res_back_attach res 0xa
virtio_gpu_cmd_res_back_attach res 0xe
virtio_gpu_cmd_res_back_attach res 0xf
virtio_gpu_cmd_res_back_attach res 0x10
virtio_gpu_cmd_res_back_attach res 0x11
virtio_gpu_cmd_res_back_attach res 0x12
virtio_gpu_cmd_ctx_submit ctx 0x3, size 74220

[max8rr8]

I've tested the driver on win10, and got a black screen. But if change the guest os to ubuntu(same qemu command line), glxgears performs well. (glxinfo shows that the renderer is NVIDIA gpu)

It seems to be an bug in virglrenderer on nvidia related to GL_PRIMITIVE_RESTART_NV. You can try to apply this diff to fix it, but it's a bit hacky solution.

[CE1CECL]

What versions of windows does it go down to? In my case, I am experimenting with https://github.com/CE1CECL/qemu-vmvga and it works with vista (no 3d yet), trying to get 3D to work right now.

[foxlet]

I tried the pre-built driver on Windows 10 22H2 and it only showed a black screen before hard-locking and resetting. Also tested the same config with Ubuntu which had working virgl.

Using -display gtk,gl=on -device virtio-vga-gl on an AMD R9 6900HS + RX 6700S host. virglrenderer was compiled from git (considering that it's already been merged in). qemu is version 7.2.4

qemu.log

[Conan-Kudo]

Can we validate that this works with Windows 7 too? It's a fairly common virtualization guest for playing older games, and it's still WDDM class.

[max8rr8]

What versions of windows does it go down to?

Current driver theoretically supports windows 8.1, but it is built for and tested only on Windows 10 22H2. But it is important to note that support for blob resources which are required to improve performance and support vulkan will require using WDDM 2 which lift minimum windows version to 10.

Can we validate that this works with Windows 7 too?

I doubt it will work with current code, it might be possible to adapt it for windows 7, but i do not have interest in doing that (though wouldn't mind if someone else will adapt code). Plus at some point driver will have to use WDDM 2 which will splitting codebases or more likely require either dropping support for older OS.

I tried the pre-built driver on Windows 10 22H2 and it only showed a black screen before hard-locking and resetting. Also tested the same config with Ubuntu which had working virgl.

Nothing seems wrong in attached qemu.log. I don't see any lines like GLSL feature level 460 that should be printed to stdout, can you also attach stdout of qemu. Although it could be issue in kernel driver then the only way to know what's wrong is to attach kernel debugger to vm.

[Torinde]

That's great @max8rr8! It seems your focus is on modern software (Win8.1/10) and you plan going upwards in support (D3D12/Vulkan/venus). In that regards - do you plan adding:

• video encoding/decoding acceleration
• lvp/llvmpipe/softpipe - fallback when no compatible GPU is present on the host or to run the graphics on a high-corecount CPU instead of its weak integrated GPU? Or if possible to combine the two (augment the performance of a weak GPU with llvmpipe running on the CPU cores)

For the retro direction, above was mentioned Win7 and in addition I think such virgl/venus GPU will be very useful also for:

• QEMU with Win9x to run JHRobotics/softgpu
• DOSbox-X with Win9x, where it may help run the driver JHRobotics/softgpu
• 86box/PCbox with Win9x, WinXP and maybe even Vista, OS/2, BeOS, etc.

[RedGreenBlue09]

I compiled and tried it, but I got the BSOD DRIVER_CORRUPTED_EXPOOL.
Host: Windows 11 Pro (10.0.22621.2070), NVIDIA driver 536.67
Guest: Windows 10 Enterprise (10.0.16299.15)

First, I compiled QEMU in MSYS2 like this:

cp /c/Program\ Files\ \(x86\)/Windows\ Kits/10/Include/10.0.22621.0/um/WinHv*\
 /ucrt64/x86_64-w64-mingw32/include/

./configure \
--target-list=x86_64-softmmu,i386-softmmu,arm-softmmu \
--cpu=x86_64 \
--enable-lto \
--enable-malloc=jemalloc \
--enable-avx2 \
--enable-dsound \
--enable-hax \
--enable-iconv \
--enable-lzo \
--enable-opengl \
--enable-png \
--enable-sdl \
--enable-sdl-image \
--enable-spice \
--enable-spice-protocol \
--enable-tcg \
--enable-whpx \
--enable-virglrenderer \
--disable-docs 

I compiled virglrenderer (upstream) like your comment above, then replaced the dll in QEMU dir with the newly built one.

Then I compiled mesa with VS (MinGW GCC prints a bunch of errors related to the WDK headers, so I gave up):

meson setup build/ -Dgallium-drivers=virgl -Dgallium-d3d10umd=true -Dgallium-wgl-dll-name=viogpu_wgl -Dgallium-d3d10-dll-name=viogpu_d3d10 --backend=vs

It outputs to my root folder (C:\) so I copied bin to a directory named mesa_prefix.
I opened cmd, set MESA_PREFIX to the mesa_prefix directory then compiled viogpu.sln with msbuild (Configuration="Win10 Release", Platform=x64).

I signed the output in viogpu\viogpu3d\objfre_win10_amd64\amd64\viogpu3d:

for /R . %a in (*.exe, *.sys, *.dll) do signtool sign /f "Surface.pfx" /t "http://timestamp.sectigo.com" /fd certHash "%a"

Then I booted up the VM, enabled test signing and then install the driver. The screen instantly went black and Windows crashes.

WinDBG, qemu logs and VM command line attached. I tried using serial kernel debug but it just hangs forever at boot so I analyzed the dump file instead.

windbg.txt qemu.log cmdline.txt

[RedGreenBlue09]

Additionally, I tried booting Linux Mint 20.2 live cd but QEMU itself crashed on some heap corruption issue. I replaced the original MSYS2 virglrenderer dll which fixed it. The Windows 10 guest BSOD remains.

[max8rr8]

Hi @RedGreenBlue09, it seems that kernel crash happened during driver unloading (VioGpu3DRemoveDevice in backtrace) and while it is a problem, it is not the root cause for driver not working on your system as for some unknown reason windows triggers driver unload.

You should try to attach windbg to running vm (try to use network debugging instead of serial, it works for me). Additionally when connecting to vm from windbg enable break on connection and run following command

bp  watchdog!WdLogEvent5_WdError "k; g"

This windbg command should add backtrace logging to all errors happening in dxgkrnl. After you ran that command continue kernel execution with windbg command g or a button in gui. Then get full log and send it there, so we can analyze what actually went wrong (why does unload happen).

[RedGreenBlue09]

Hi @RedGreenBlue09, it seems that kernel crash happened during driver unloading (VioGpu3DRemoveDevice in backtrace) and while it is a problem, it is not the root cause for driver not working on your system as for some unknown reason windows triggers driver unload.

You should try to attach windbg to running vm (try to use network debugging instead of serial, it works for me). Additionally when connecting to vm from windbg enable break on connection and run following command

bp  watchdog!WdLogEvent5_WdError "k; g"

This windbg command should add backtrace logging to all errors happening in dxgkrnl. After you ran that command continue kernel execution with windbg command g or a button in gui. Then get full log and send it there, so we can analyze what actually went wrong (why does unload happen).

I will test again later. Also, if I use VGA + virtio-gpu-gl setup the driver don't load and it have error 49 in device manager. The error message is something like "Windows unloaded the driver because it has reported problems"

[RedGreenBlue09]

... Network debug also hangs like serial. Interestingly that doesn't happen with ReactOS.

[RedGreenBlue09]

I have no luck with live kernel debug. I tried using accel tcg, official qemu, none of these helps. If I enable boot debug, Windows boot manager itself freezes. Absolutely no idea.

Edit: Even ditching OVMF for SeaBIOS, it still freezes.

[RedGreenBlue09]

@max8rr8 Okay, I'm sorry for the rant. It doesn't hang forever, just 30 minutes :((. I tried your command and got an error:

0: kd> bp  watchdog!WdLogEvent5_WdError "k; g"
Bp expression 'watchdog!WdLogEvent5_WdError ' could not be resolved, adding deferred bp

[RedGreenBlue09]

After reloading symbols, the error is gone. Here is the new log.

windbg2.txt

[cdm1993dk]

Hi, I tried this driver and used device management to install viogpu3d.inf, however PAGED_CODE() in VioGpu3DSubmitCommand() failed, any suggestions? I am using win10 22H2 as guest.

[DemiMarie]

@max8rr8 do you plan on supporting “seamless” mode, where each window in the guest is a window on the host?

[max8rr8]

Hi, I tried this driver and used device management to install viogpu3d.inf, however PAGED_CODE() in VioGpu3DSubmitCommand() failed, any suggestions? I am using win10 22H2 as guest.

Hi, seems that i missed some left over paged code in non-paged context. It should be fixed now (at least for SubmitCommand)

@max8rr8 do you plan on supporting “seamless” mode, where each window in the guest is a window on the host?

While i definitely would like to support seamleess mode, it's out of scope of 3d driver itself. I see two possible solutions:

1. Use RDP like WSL-g does. This impacts performance (not that critically though) but probably has higher integration with guest system.
2. Implement virtio-wayland for windows. This will provide high perfromance (as through virtio-wayland there is ability to share 3d resources from guest with host) and probably will integrate better with host os. On the other side it is harder to implement compared to first option and probably will integrate with guest os worse than first option.

I personally prefer 2nd solution, and might even try to implement it at some point in future.

[DemiMarie]

@max8rr8 do you plan on supporting “seamless” mode, where each window in the guest is a window on the host?

While i definitely would like to support seamleess mode, it's out of scope of 3d driver itself.

A proper implementation of seamless mode requires some support from the driver so that the individual window buffers can be extracted. Without this, there can be problems due to mismatched window stacking order between the guest and the host, especially with Wayland where the host’s window stacking order is not visible.

I see two possible solutions:

1. Use RDP like WSL-g does. This impacts performance (not that critically though) but probably has higher integration with guest system.
2. Implement virtio-wayland for windows. This will provide high perfromance (as through virtio-wayland there is ability to share 3d resources from guest with host) and probably will integrate better with host os. On the other side it is harder to implement compared to first option and probably will integrate with guest os worse than first option.

I personally prefer 2nd solution, and might even try to implement it at some point in future.

That would be amazing.

[RedGreenBlue09]

Hi @max8rr8, I retested the latest update of both mesa and viogpu3d. Seems like the number of errors has reduced, but it's still not working. I can see and move the mouse cursor for a moment (not entirely black like previously), then driver unloaded; something is double deleted & kernel crashed.

windbg_new.txt

Some of the below information might be related:

• I'm using virtio-vga-gl not virtio-gpu-gl.
• Virglrenderer 1.0.0 doesn't compile on Windows anymore so I'm using the one from August 18.
• Build 10.0.16299 might be old?

[he3als]

Is it possible to compile/test this for aarch64?

[RedGreenBlue09]

BSODs are fixed with the latest changes. Still, driver is unloaded (code 43).

[leo60228]

The prebuilt link is giving a 404, is there an updated link?

[awsms]

+1 for the pre-built binaries

[RedGreenBlue09]

For someone who need prebuilt binaries: viogpu3d.zip
This is signed driver compiled with MSVC. It unloads in my VM but I don't think the driver itself is the problem.

I also have debug symbols, not included here.

[lyc8503]

For someone who need prebuilt binaries: viogpu3d.zip This is signed driver compiled with MSVC. It unloads in my VM but I don't think the driver itself is the problem.

I also have debug symbols, not included here.

Had a very rough test on Windows 11 22H2 Build 22621.1992(guest) and latest PVE 8 virtio-gl (hypervisor)

It boots but just displays a completely black screen, RDP connects but also displays a black screen. (at least Internet works and not encounter BSODs)

Inexperienced in Windows kernel dev and not able to help much, but looking forward to your great work!

[AnErrupTion]

For someone who need prebuilt binaries: viogpu3d.zip This is signed driver compiled with MSVC. It unloads in my VM but I don't think the driver itself is the problem.

I also have debug symbols, not included here.

For some reason, I can't even install it, it says: "A problem was encountered while attempting to add the driver to the store.", do you perhaps know why?