kvm-guest-drivers-windows
kvm-guest-drivers-windows copied to clipboard
virtio-win
BSOD with virtio-scsi
Describe the bug When running Passmark Performance Test, the vm ends with BSOD and error pfn_list_corrupt or critical_process_died. After changing the disks to sata controller, the test finished without errors.
Tested also with virtio-win-0.1.229.
The disks are configured like this:
<disk type="block" device="disk">
<driver name="qemu" type="raw" cache="none" io="native" discard="unmap"/>
<source dev="/dev/zvol/rpool/vms/vm1/system"/>
<target dev="sda" bus="scsi"/>
<address type="drive" controller="0" bus="0" target="0" unit="0"/>
</disk>
<controller type="scsi" index="0" model="virtio-scsi">
<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</controller>
I also have vfio gpu passthrough in this vm.
OS: archlinux
QEmu: 7.2.0-3
Any idea about this problem ?
To Reproduce Steps to reproduce the behaviour:
Expected behavior Shoud not crash
Screenshots If applicable, add screenshots to help explain your problem.
Host:
- Disto: archlinux
- Kernel 6.1.8-arch1-1
- QEMU 7.2.0-3
- QEMU command line
- libvirt 1:9.0.0-1
- libvirt XML file
VM:
- Windows 11
- Which driver has a problem virtio-scsi
- Driver version or commit hash that was used to build the driver 100.92.104.22900
Additional context Add any other context about the problem here.
@vinod-melarkode Thank you for reporting the issue. Is there any way you can share the crash dump file?
Best, Vadim.
Unfortunately is not possible because is not generated. On the BSOD page it only displays 0% progress.
Hi @vinod-melarkode
If you can reproduce the bug, I suggest you to do the following:
- Add
<vmcoreinfo state='on'/> - Install
FWCfgdriver on the guest side if it is not installed yet - When BSoD occurs, create a dump with QEMU monitor command
dump-guest-memory -wfrom the host side
If it is too hard, you can share with us QEMU ELF dump produced by dump-guest-memory
Hi,
can you send me a gpg key in order to encrypt the dumps ?
Thanks.
Hi,
can you send me a gpg key in order to encrypt the dumps ?
Hi @vrozenfe, could you please help?
@vrozenfe Hello, I have the same problem. May I ask what your host machine is AMD or Intel?
@w123456w30w it's Intel [vrozenfe@virtowin qemu]$ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 46 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 40 On-line CPU(s) list: 0-39 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz CPU family: 6 Model: 85 Thread(s) per core: 2 Core(s) per socket: 10 Socket(s): 2 Stepping: 7 CPU(s) scaling MHz: 32% CPU max MHz: 3200.0000 CPU min MHz: 1000.0000 BogoMIPS: 4800.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fx sr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts re p_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_t imer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_ single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm mpx rdt_a avx512f avx512dq rdse ed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_clear flush_l1d arch_capabilities Virtualization features: Virtualization: VT-x Caches (sum of all): L1d: 640 KiB (20 instances) L1i: 640 KiB (20 instances) L2: 20 MiB (20 instances) L3: 27.5 MiB (2 instances) NUMA: NUMA node(s): 2 NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39 Vulnerabilities: Itlb multihit: KVM: Mitigation: Split huge pages L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Retbleed: Mitigation; Enhanced IBRS Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence Srbds: Not affected Tsx async abort: Mitigation; TSX disabled
But for the sake of testing we can ask our QE to reproduce the issue on both Intel and AMD.
Best, Vadim.
Hi @vinod-melarkode,
I tried to reproduce this issue on my host, but unfortunately, I didn't reproduce it, I run the PASSMARK test finished smoothly, no hit BSOD, Could I ask some more details for this issue?
-
I used a emulated block disk /dev/sdb to replace your used "/dev/zvol/rpool/vms/vm1/system"~ I'm not sure if this issue is related the hard disk devices, Could you tell the detail about this volume "/dev/zvol/rpool/vms/vm1/system" info? The disk size, and the disk type? SAS or SATA or others? or if it's a physical hard disk or a emulated on?
-
Which BIOS mode was used for your vm? Legacy or OVMF? I used OVMF this reproduce test.
-
And could you help to provide your used libvirt xml file(includes memory/cpu with flag info) or qemu-kvm commands?
I used parts of xml file as follows:
virsh dumpxml windows11
I used RHEL9, versions as forllows:
- kernel-5.14.0-249.el9.x86_64
- qemu-kvm-7.2.0-8.el9.x86_64
- VM: Win11
- edk2-ovmf-20221207gitfff6d81270b5-6.el9.noarch
- virtio-win-1.9.32-0.el9_1.iso
Thanks~ Peixiu
Hi,
- /dev/zvol/rpool/vms/vm1/system is a ZFS zvol
- OVMF
- I have attached my xml file.
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>vm1</name>
<uuid>11111111-1111-1111-1111-111111111111</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/10"/>
</libosinfo:libosinfo>
</metadata>
<memory unit='KiB'>16777216</memory>
<currentMemory unit='KiB'>16777216</currentMemory>
<memoryBacking>
<hugepages/>
<nosharepages/>
<locked/>
</memoryBacking>
<vcpu placement='static'>16</vcpu>
<os>
<type arch='x86_64' machine='pc-q35-7.2'>hvm</type>
<loader readonly='yes' secure='yes' type='pflash'>/usr/share/edk2-ovmf/x64/OVMF_CODE.secboot.4m.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/win11_VARS.fd</nvram>
</os>
<features>
<acpi/>
<apic/>
<hyperv mode='custom'>
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
<vpindex state='on'/>
<runtime state='on'/>
<synic state='on'/>
<stimer state='on'/>
<reset state='on'/>
<vendor_id state='on' value='1234567890ab'/>
<frequencies state='on'/>
<reenlightenment state='on'/>
<tlbflush state='on'/>
<ipi state='on'/>
<evmcs state='off'/>
</hyperv>
<kvm>
<hidden state='on'/>
<hint-dedicated state='on'/>
<poll-control state='on'/>
</kvm>
<vmport state='off'/>
<smm state='on'/>
</features>
<cpu mode='host-passthrough' check='none' migratable='off'>
<topology sockets='1' dies='1' cores='8' threads='2'/>
<cache mode='passthrough'/>
<feature policy='require' name='topoext'/>
<feature policy='disable' name='svm'/>
<feature policy='disable' name='x2apic'/>
<feature policy='require' name='hypervisor'/>
<feature policy='require' name='invtsc'/>
</cpu>
<clock offset='timezone' timezone='Europe/Bucharest'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
<timer name='kvmclock' present='no'/>
<timer name='hypervclock' present='yes'/>
<timer name='tsc' present='yes' mode='native'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native' discard='unmap' detect_zeroes='unmap'/>
<source dev='/dev/zvol/rpool/srv/vms/vm1/system'/>
<target dev='sda' bus='scsi'/>
<boot order='2'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native' discard='unmap' detect_zeroes='unmap'/>
<source dev='/dev/zvol/rpool/srv/vms/vm1/apps'/>
<target dev='sdb' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</controller>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='1' port='0x10'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='2' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='2' port='0x11'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='3' port='0x12'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
</controller>
<controller type='pci' index='4' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='4' port='0x13'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
</controller>
<controller type='pci' index='5' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='5' port='0x14'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
</controller>
<controller type='pci' index='6' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='6' port='0x15'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
</controller>
<controller type='pci' index='7' model='pcie-to-pci-bridge'>
<model name='pcie-pci-bridge'/>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='8' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='8' port='0x8'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
</controller>
<controller type='pci' index='9' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='9' port='0x9'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
</controller>
<controller type='pci' index='10' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='10' port='0xa'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='11' model='pcie-root-port'>
<model name='pcie-root-port'/>
<target chassis='11' port='0xb'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</controller>
<controller type='scsi' index='0' model='virtio-scsi'>
<address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:00:00:71'/>
<source bridge='lan-br'/>
<target dev='lan-win11'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<tpm model='tpm-crb'>
<backend type='emulator' version='2.0'/>
</tpm>
<audio id='1' type='none'/>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x0d' slot='0x00' function='0x0'/>
</source>
<rom file='/srv/vms/deps/roms/Inno3D_patched.rom'/>
<address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x0d' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='yes'>
<source>
<vendor id='0x046d'/>
<product id='0xc08b'/>
</source>
<address type='usb' bus='0' port='2'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x10' slot='0x00' function='0x3'/>
</source>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='yes'>
<source>
<vendor id='0x046d'/>
<product id='0xc52b'/>
</source>
<address type='usb' bus='0' port='3'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='yes'>
<source>
<vendor id='0x8087'/>
<product id='0x0029'/>
</source>
<address type='usb' bus='0' port='1'/>
</hostdev>
<memballoon model='none'/>
</devices>
<qemu:capabilities>
<qemu:del capability='usb-host.hostdevice'/>
</qemu:capabilities>
</domain>
Thanks.
Hi @vinod-melarkode,
Sorry for reply late, I tried to test with zfs zvol block device on a RHEL9.2 host, the steps as follows, but I also did not reproduce this issue. Could you help to have a look? thanks~
The reproduced steps :
-
On a RHEL9.2 Host, configure zfs zvol first.
-
dnf install -y https://zfsonlinux.org/epel/zfs-release-2-2$(rpm --eval "%{dist}").noarch.rpm
-
dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
-
dnf install -y epel-release
-
dnf install -y kernel-devel
-
dnf install -y zfs
-
modprobe zfs
-
lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 893.3G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 892.2G 0 part ├─rhel_dell--per750--34-root 253:0 0 70G 0 lvm / ├─rhel_dell--per750--34-swap 253:1 0 31.2G 0 lvm [SWAP] └─rhel_dell--per750--34-home 253:2 0 791.1G 0 lvm /home sdb 8:16 0 30G 0 disk sdc 8:32 0 30G 0 disk
-
zpool create peixiu /dev/sdb /dev/sdc
-
zpool status pool: peixiu state: ONLINE config:
NAME STATE READ WRITE CKSUM peixiu ONLINE 0 0 0 sdb ONLINE 0 0 0 sdc ONLINE 0 0 0
errors: No known data errors 11. zfs list NAME USED AVAIL REFER MOUNTPOINT peixiu 110K 57.2G 24K /peixiu 12. zfs create -V 30G peixiu/system_disk 13. zfs create -V 10G peixiu/data_disk 14. zfs list NAME USED AVAIL REFER MOUNTPOINT peixiu 41.3G 15.9G 24K /peixiu peixiu/data_disk 10.3G 26.2G 12K - peixiu/system_disk 30.9G 46.8G 12K -
- fdisk -l Disk /dev/zd0: 30 GiB, 32212254720 bytes, 62914560 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 8192 bytes I/O size (minimum/optimal): 8192 bytes / 8192 bytes
Disk /dev/zd16: 10 GiB, 10737418240 bytes, 20971520 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 8192 bytes I/O size (minimum/optimal): 8192 bytes / 8192 bytes
- Use /dev/zd0 and /dev/zd16 as vm's system disk and data disk. boot the vm up with win11.xml:
- In the Windows 11 guest, install PerformanceTest application, and run PASSMARK Tests:
- Tried a few times, did not hit BSOD.
- I tested with 100.92.104.229 and 100.92.104.228, both did not reproduce this bug.
Used version: kernel-5.14.0-285.el9.x86_64 qemu-kvm-7.2.0-11.el9_2.x86_64 virtio-win-0.1-229.iso libvirt-9.0.0-7.el9.x86_64 edk2-ovmf-20221207gitfff6d81270b5-8.el9_2.noarch
I did not add vfio gpu here, since I run on an AMD host, windows does not support iommu on AMD host. Could you help to check if my steps match your tests?
And forgive me for that I also have some other questions for this issue? How reproducible ratio for your tests? How about for the latest virtio-win 235 build? Did you reproduce it without vfio gpu before? If possible, could you help to have a try without vfio gpu device?
Thanks a lot~ Peixiu