kvm-guest-drivers-windows icon indicating copy to clipboard operation
kvm-guest-drivers-windows copied to clipboard

viogpudo /virtio-vga broken in recent Windows builds

Open kroese opened this issue 1 year ago • 56 comments

In recent Windows versions, like Windows 11 IoT 24H2, when using the virtio-vga graphics device, the screen turns black when installing the viogpudo driver and stays black until the next reboot.

Using the std VGA device everything works.

The problem even exists with the new beta drivers, version 0.1.258.

kroese avatar Jun 06 '24 08:06 kroese

@kroese Please provide QEMU CLI, QE can't reproduce this issue.

image

kostyanf14 avatar Jun 06 '24 10:06 kostyanf14

Hi @kroese,

could you please provide more details like when you installed the driver? during the installation of system or installed virtio-vga after installing system. And yes, please provide us the qemu-cli.

Thanks Dehan Meng

6-dehan avatar Jun 06 '24 13:06 6-dehan

@6-dehan The commandline is:

-nodefaults
-cpu host,kvm=on,l3-cache=on,migratable=no,+hypervisor,hv_passthrough,+invtsc
-smp 2,sockets=1,dies=1,cores=2,threads=1
-m 4G
-machine type=q35,smm=off,graphics=off,vmport=off,dump-guest-core=off,hpet=off,accel=kvm
-enable-kvm
-global kvm-pit.lost_tick_policy=discard
-display vnc=:0,websocket=5700
-vga virtio
-monitor telnet:localhost:7100,server,nowait,nodelay
-daemonize
-D /run/shm/qemu.log
-pidfile /run/shm/qemu.pid
-name windows,process=windows,debug-threads=on
-serial pty
-device qemu-xhci,id=xhci
-device usb-tablet
-netdev tap,ifname=qemu,script=no,downscript=no,id=hostnet0
-device virtio-net-pci,romfile=,netdev=hostnet0,mac=02:BA:F0:B0:CC:13,id=net0
-drive file=/storage/windows.iso,id=cdrom0,format=raw,readonly=on,media=cdrom,if=none
-device virtio-scsi-pci,id=cdrom0b,bus=pcie.0,addr=0x5,iothread=io2
-device scsi-cd,drive=cdrom0,bus=cdrom0b.0,bootindex=10
-drive file=/storage/data.img,id=data3,format=raw,cache=none,aio=native,discard=on,detect-zeroes=on,if=none
-device virtio-scsi-pci,id=data3b,bus=pcie.0,addr=0xa,iothread=io2
-device scsi-hd,drive=data3,bus=data3b.0,channel=0,scsi-id=0,lun=0,rotation_rate=1,bootindex=3
-object iothread,id=io2
-rtc base=localtime
-global ICH9-LPC.disable_s3=1
-global ICH9-LPC.disable_s4=1
-drive file=/storage/windows.rom,if=pflash,unit=0,format=raw,readonly=on
-drive file=/storage/windows.vars,if=pflash,unit=1,format=raw
-object rng-random,id=objrng0,filename=/dev/urandom
-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pcie.0,addr=0x1c

it happens when the driver is installed during the setup (automaticly via unattended install, or when loading it manually when windows asks for boot-critical drivers).

  • When changing the vga device from virtio to std, it does not occur.
  • When removing the viogpudo folder from the drivers, it does not occur.
  • When changing the ISO from 24H2 to older Windows versions ( 23H2 and lower) it does not occur.

kroese avatar Jun 06 '24 14:06 kroese

Thanks @kroese , After investigating your qemu-cli with @kostyanf14 , we found the 'smm=off' should change to on and all 'scsi-hd' should change to IDE. then virtio-vga works. so maybe virtio-scsi has some problem. image

6-dehan avatar Jun 07 '24 09:06 6-dehan

@6-dehan Thanks! But it is on purpose that I did not enable Secure Boot (smm=off), because there are other problems (not related to virtio-win) when enabling Secure Boot on hosts with certain CPU models. Also I reported before that vioscsi is having problems, see https://github.com/virtio-win/kvm-guest-drivers-windows/issues/1100 . But they said it was not a virto-win issue.

I am using these drivers in my project ( https://github.com/dockur/windows ) which has a large userbase, so I cannot just switch the whole project to IDE or Secure Boot, because it will affect thousands of users. I guess I have no other option to wait until the drivers for 24H2 are more mature, before I can offer those operating systems as an option.

kroese avatar Jun 07 '24 11:06 kroese

Problem still present in v0.1.262

@6-dehan Also it does not seem to be related to having smm=off or smm=on, it occurs in both cases for me. And both with scsi or ide disks.

The only new thing I discovered is that if you just wait a couple minutes with the black screen, the image comes back and the installation will actually proceed.

kroese avatar Aug 17 '24 23:08 kroese

Hi @kroese, I could reproduce this issue on

  1. win2025/Win11 24H2/win2022 with the latest 262prewhql.iso.
  2. win2025/Win11 24H2 with the 258prewhql.iso

scenario:

  • After loading virtio-GPU driver. the screen gotta black situation and couldn't login system.

6-dehan avatar Aug 20 '24 02:08 6-dehan

@6-dehan

Do we have a Jira issue for tracing this problem? Thanks, Vadim.

vrozenfe avatar Aug 20 '24 04:08 vrozenfe

@vrozenfe , yeah, I'll paste issue here later. I was testing for more details update.

6-dehan avatar Aug 20 '24 05:08 6-dehan

Update: If use the signed driver like 'virtio-win-1.9.40-0.el9_4.iso', the system and vio-GPU driver will be installed successfully without any black screen kind of issue. so I'll try the latest virtio-win rpm after the @vrozenfe makes it.

6-dehan avatar Aug 20 '24 06:08 6-dehan

@vrozenfe @kroese Update the latest result: I tried draft virtio-win.iso (not release, test-only) 2025/win11 24h2/2022 will not hit any black screen problem. anyway we have this issue ‘RHEL-55254’ for tracking this problem. feel free to contact us if needed. thanks

6-dehan avatar Aug 20 '24 09:08 6-dehan

@6-dehan Is there anywhere I can download that ISO to confirm if it solves the problem?

kroese avatar Aug 20 '24 20:08 kroese

@6-dehan Is there anywhere I can download that ISO to confirm if it solves the problem?

@kroese That's a draft build for internal testing. But no worry for sure, it will be published soon as our latest virtio-win.iso.

6-dehan avatar Aug 21 '24 02:08 6-dehan

@6-dehan Since a couple of weeks have passed without new ISO, out of curiosity I tried it with the virtio-win-1.9.40-0.el9_4.iso from RockyLinux/CentOS.

But I still got the black screen both with Server 2025 (and the W11 driver) and Windows 11 24H2 IoT (with the W11 driver).

So either the 1.9.40 drivers in the ISO from RockyLinux/CentOS are not signed?? Or your test was not correct?

kroese avatar Sep 12 '24 16:09 kroese

@kroese Please ping me at vrozenfe_at_redhat_dot_com Best, Vadim.

vrozenfe avatar Sep 13 '24 02:09 vrozenfe

@6-dehan Since a couple of weeks have passed without new ISO, out of curiosity I tried it with the virtio-win-1.9.40-0.el9_4.iso from RockyLinux/CentOS.

But I still got the black screen both with Server 2025 (and the W11 driver) and Windows 11 24H2 IoT (with the W11 driver).

So either the 1.9.40 drivers in the ISO from RockyLinux/CentOS are not signed?? Or your test was not correct?

strange, I can get installation pass for both of them with signed iso without black screen, even yours virtio-win-1.9.40-0.el9_4.iso image

6-dehan avatar Sep 13 '24 03:09 6-dehan

@6-dehan But are you installing them AFTER Windows is already installed? Because it looks that way from your screenshot? My whole issue is when installing them DURING the Windows installation (unattendedly), not when manually installing them afterwards.

kroese avatar Sep 17 '24 20:09 kroese

@6-dehan But are you installing them AFTER Windows is already installed? Because it looks that way from your screenshot? My whole issue is when installing them DURING the Windows installation (unattendedly), not when manually installing them afterwards.

Surely I installed the driver during the Windows installation. I just provided you with a screenshot of the successful installation.

6-dehan avatar Sep 18 '24 01:09 6-dehan

@6-dehan From the screenshot it appears to be Windows Server 2025. Currently I also cannot reproduce it with Server 2025 anymore. Maybe I was confused (because I have some other issues with 2025 like the scsi driver failing) but all my testing recently was with: Windows 11 24H2 (IoT Enterprise LTSC).

And there I can reproduce the issue 100 percent of the time, and with every version of the driver (0.1.262, 1.9.43, the .zip Vadim send me, etc).

So if its not too much to ask, could you please repeat your test with Win11 IoT / LTSC.

kroese avatar Sep 18 '24 01:09 kroese

@6-dehan From the screenshot it appears to be Windows Server 2025. Currently I also cannot reproduce it with Server 2025 anymore. Maybe I was confused (because I have some other issues with 2025 like the scsi driver failing) but all my testing recently was with: Windows 11 24H2 (IoT Enterprise LTSC).

And there I can reproduce the issue 100 percent of the time, and with every version of the driver (0.1.262, 1.9.43, the .zip Vadim send me, etc).

So if its not too much to ask, could you please repeat your test with Win11 IoT / LTSC.

No problem, I'd like to reproduce it on Win11. I'll update results later.

6-dehan avatar Sep 18 '24 01:09 6-dehan

Hi @kroese, The iso should be this kind of iso right ? image

6-dehan avatar Sep 18 '24 06:09 6-dehan

@6-dehan Yes!!

kroese avatar Sep 18 '24 08:09 kroese

@6-dehan Yes!!

Hi @kroese , The issue is definitely there. I tried:

  • virtio-win-1.9.40-0.el9_4.iso (win10 folder, because the system can't load GPU driver within win11 folder)
  • virtio-win-1.9.43-0.el9_5.iso (win11 folder)

But we don't test this iso in our matrix.

6-dehan avatar Sep 18 '24 09:09 6-dehan

@6-dehan A few days ago, Microsoft released the final release of Windows 11 LTSC:

https://oemsoc.download.prss.microsoft.com/dbazure/X23-81951_26100.1742.240906-0331.ge_release_svc_refresh_CLIENT_ENTERPRISES_OEM_x64FRE_en-us.iso_640de540-87

Is the LTSC ISO in your matrix now, and did you already find the cause for the black screen?

It also happens on non-LTSC builds, like the official 24H2:

https://software.download.prss.microsoft.com/dbazure/Win11_24H2_English_x64.iso?t=104abaf5-0027-4b07-a969-8734a6d49047&P1=1727879576&P2=601&P3=2&P4=YAYJdSfedkbAki1J6f6e9qncG0lt%2buZ7rfLtMlXOAqTrILhglaS7gWe10%2b%2bBUMMkDIoQNrba%2f6i9Z88sYAGrVJveIOjiNxd%2br9lTU7saH1iammeWZCHSdSRFlGWyR8M2x2JotjnfxSxny2JVYN4OK6qJqKFecPDs5g3sgumACcmqsIxeVYWeG8zjTnCtEDdMm6q5%2fwPQZGH60jWnWe7gaD2642w8ZgNSTSifA0jQ0pKOtBh2vXeDeob6m%2fbENu%2bI8fdoJnSlRFK%2fB%2bT9AW0i0LPM79dHP25O%2bz%2fAPrLrxlOMHm2%2bxPZfyTZDEl%2b56%2fdk2v51o9NznmOKaXoWm5pIoQ%3d%3d

And also on some Windows Server 2025 builds, like:

https://dl.bobpony.com/windows/server/2025/en-us_windows_server_2025_preview_x64_dvd_ce9eb1a5.iso

So that are already three different Windows editions that are affected.

kroese avatar Oct 01 '24 15:10 kroese

@6-dehan Did you see this issue during certification for Server 2025?

kostyanf14 avatar Oct 01 '24 15:10 kostyanf14

@kroese We normally use Windows images distributed via the official Microsoft channels like MSDN subscription. We do use images distributed via Microsoft Partner Centre Portal Canary channel and Windows Insider beta channel for preview testing. But as I said our official WHQL and functional testing is always done on officially released OEM/GA Windows releases .

All the best, Vadim.

vrozenfe avatar Oct 02 '24 23:10 vrozenfe

@vrozenfe These images are not insider/beta builds? The 24H2 is the official release that you get as a consumer when you go to:

https://www.microsoft.com/en-us/software-download/windows11

The LTSC is also the official/final release.

If this issue only happened on previews/betas, I would have understand that it would get little priority. But 24H2 is officially released, and its the only version you can download as a consumer from the Microsoft site right now.

kroese avatar Oct 03 '24 03:10 kroese

@kroese

Let me check it on IoT LTSC image 26100.1.240331-1435.ge_release_CLIENT_IOT_LTSC_EVAL_x64FRE_en-us.iso available from MSDN subscription download first.

Best, Vadim.

vrozenfe avatar Oct 03 '24 04:10 vrozenfe

Didn't have any problem installing viogpudo driver on 26100.1.240331-1435.ge_release_CLIENT_IOT_LTSC_EVAL_x64FRE_en-us.iso

(qemu) info version 8.2.50v8.2.0-645-g977542ded7

[vrozenfe@milly tmp]$ uname -a Linux milly 6.5.12-100.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Nov 20 22:28:44 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[vrozenfe@milly tmp]$ sudo ps -aux | grep qemu root 84827 0.0 0.0 236304 9088 pts/3 S+ 16:36 0:00 sudo /home/vrozenfe/work/upstream/qemu/build/x86_64-softmmu/qemu-system-x86_64 -name WIN11-24H2 -machine q35 -nodefaults -vga virtio -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x3 -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x3.0x1 -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x3.0x2 -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x3.0x3 -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x3.0x4 -device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x3.0x5 -device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x3.0x6 -device pcie-root-port,port=0x17,chassis=8,id=pci.8,bus=pcie.0,addr=0x3.0x7 -device virtio-net-pci,mac=9a:36:83:b6:3d:05,id=idJVpmsF,netdev=id23ZUK6,bus=pci.3 -netdev tap,id=id23ZUK6,vhost=on,script=/etc/qemu-ifup,downscript=no -device usb-ehci,id=ehci0,bus=pci.5 -m 4G -smp 2,maxcpus=4 -cpu host,hv_stimer,hv_synic,hv_vpindex,hv_reset,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv-tlbflush,+kvm_pv_unhalt -drive id=drive_cd0,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/vrozenfe/work/isos/26100.1.240331-1435.ge_release_CLIENT_IOT_LTSC_EVAL_x64FRE_en-us.iso -device ide-cd,id=cd0,drive=drive_cd0,bus=ide.0,unit=0,bootindex=0 -drive id=drive_cd1,if=none,snapshot=off,aio=threads,cache=none,media=cdrom,file=/home/vrozenfe/work/isos/virtio-win-1.9.43.iso -device ide-cd,id=cd1,drive=drive_cd1,bus=ide.1,unit=0 -device piix3-usb-uhci,id=usb -device usb-tablet,id=input0 -rtc base=localtime,clock=host,driftfix=slew -boot order=cdn,once=c,menu=off,strict=off -enable-kvm -qmp tcp:0:1232,server,nowait -monitor stdio -drive file=/usr/share/edk2/ovmf/OVMF_CODE.secboot.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/home/vrozenfe/work/vms/24h2/vars.fd,if=pflash,format=raw,unit=1 -chardev socket,id=chrtpm,path=/tmp/mytpm1/swtpm-sock -tpmdev emulator,id=tpm0,chardev=chrtpm -device tpm-tis,tpmdev=tpm0 -blockdev node-name=file_stg2,driver=file,cache.direct=on,cache.no-flush=off,filename=/home/vrozenfe/work/images/w11_24h2.qcow2,aio=threads -blockdev node-name=drive_stg2,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_stg2 -device virtio-blk-pci,id=stg2,drive=drive_stg2,bootindex=1 -drive file=/home/vrozenfe/work/vms/24h2/uefi.raw,if=none,id=drive-usb-2-0,media=disk,format=raw,cache=none,werror=stop,rerror=stop,aio=threads -device usb-storage,bus=ehci0.0,drive=drive-usb-2-0,id=usb-2-0,removable=on

Screenshot from 2024-10-03 16-39-46

Going to check https://www.microsoft.com/en-us/software-download/windows11 now.

vrozenfe avatar Oct 03 '24 06:10 vrozenfe

@vrozenfe The issue is when the GPU driver is installed automaticly (unattendly) during installation (via an XML answer file or in the $WinPEDriver$ folder). So is that what you tested, or did you install them manually AFTER Windows was already installed? Also, @6-dehan was able to reproduce it with that same ISO?

kroese avatar Oct 03 '24 07:10 kroese