qubes-issues Boot loop from QubesOS 4.2.4 ISO under nested virtualization on INTEL CPUs

Qubes OS release

Qubes OS 4.2

Brief summary

Boot loop when booting on QubesOS 4.2.4 ISO in Nested Virtualization on INTEL.

Steps to reproduce

Install VMware Workstation (e.g. 17.6.3) on a Windows 10 PC with INTEL CPU (e.g. i7 10th gen) and support of nested virtualization
Create a VM with processor option Virtualize Intel VT-x/EPT or AMD-V/RVI enabled
try to boot to install QubesOS => boot loop

Expected behavior

normal boot

Actual behavior

boot loop

Additional information

This would came from a regression introduced in XEN 4.7.3 and fixed by this commit - https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=dd05d265b8abda4cc7206b29cd71b77fb46658bf

May 14 '25 19:05 ohault

Nested virtualization is not supported on Qubes OS, so this may not be considered a bug in Qubes OS. However, since there's already an upstream commit that fixes it, I'll wait to see what the devs think.

May 14 '25 22:05 andrewdavidwong

QubesOS 4.1.2 does not have this issue.

I'm currently testing with

QubesOS 4.2.3
QubesOS 4.2.2
QubesOS 4.2.1
QubesOS 4.2.0

May 15 '25 14:05 ohault

I'd recommend gathering a bit of debug information, so as to make sure it's indeed a Xen-related regression, rather than trying to load an unsupported graphics driver of the VMware paravirtualized GPU.

May 15 '25 15:05 aronowski

I'd recommend gathering a bit of debug information, so as to make sure it's indeed a Xen-related regression, rather than trying to load an unsupported graphics driver of the VMware paravirtualized GPU.

I can easily log serial outputs of the VM to a file, but how to enable logging to serial for the QubesOS ISO, or another way?

May 15 '25 15:05 ohault

how to enable logging to serial for the QubesOS ISO

In grub, add com1=115200,8n1 console=com1 to xen.gz parameters

May 15 '25 16:05 marmarek

 Xen 4.17.5
(XEN) Xen version 4.17.5 (mockbuild@[unknown]) (gcc (GCC) 12.3.1 20230508 (Red Hat 12.3.1-1)) debug=n Sun Feb 16 03:41:49 GMT 2025
(XEN) Latest ChangeSet: 
(XEN) Bootloader: GRUB 2.06
(XEN) Command line: com1=115200,8n1 console=com1
(XEN) Xen image load base address: 0xbf400000
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN) Disc information:
(XEN)  Found 0 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  [0000000000000000, 0000000000097bff] (usable)
(XEN)  [0000000000097c00, 000000000009ffff] (reserved)
(XEN)  [00000000000ce000, 00000000000cffff] (reserved)
(XEN)  [00000000000dc000, 00000000000fffff] (reserved)
(XEN)  [0000000000100000, 00000000bfecffff] (usable)
(XEN)  [00000000bfed0000, 00000000bfefefff] (ACPI data)
(XEN)  [00000000bfeff000, 00000000bfefffff] (ACPI NVS)
(XEN)  [00000000bff00000, 00000000bfffffff] (usable)
(XEN)  [00000000f0000000, 00000000f7ffffff] (reserved)
(XEN)  [00000000fec00000, 00000000fec0ffff] (reserved)
(XEN)  [00000000fee00000, 00000000fee00fff] (reserved)
(XEN)  [00000000fffe0000, 00000000ffffffff] (reserved)
(XEN)  [0000000100000000, 000000023fffffff] (usable)
(XEN) ACPI: RSDP 000F6A00, 0024 (r2 PTLTD )
(XEN) ACPI: XSDT BFEDB633, 005C (r1 INTEL  440BX     6040000 VMW   1324272)
(XEN) ACPI: FACP BFEFEE73, 00F4 (r4 INTEL  440BX     6040000 PTL     F4240)
(XEN) ACPI: DSDT BFEDD001, 21E72 (r1 PTLTD  Custom    6040000 MSFT  3000001)
(XEN) ACPI: FACS BFEFFFC0, 0040
(XEN) ACPI: BOOT BFEDCFB4, 0028 (r1 PTLTD  $SBFTBL$  6040000  LTP        1)
(XEN) ACPI: APIC BFEDC872, 0742 (r1 PTLTD  	 APIC    6040000  LTP        0)
(XEN) ACPI: MCFG BFEDC836, 003C (r1 PTLTD  $PCITBL$  6040000  LTP        1)
(XEN) ACPI: SRAT BFEDB72F, 08D0 (r2 VMWARE MEMPLUG   6040000 VMW         1)
(XEN) ACPI: HPET BFEDB6F7, 0038 (r1 VMWARE VMW HPET  6040000 VMW         1)
(XEN) ACPI: WAET BFEDB6CF, 0028 (r1 VMWARE VMW WAET  6040000 VMW         1)
(XEN) System RAM: 8191MB (8387996kB)
(XEN) Domain heap initialised
(XEN) IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23
(XEN) Switched to APIC driver x2apic_phys
(XEN) ----[ Xen-4.17.5  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82d040281311>] intel.c#init_intel+0x51/0x670
(XEN) RFLAGS: 0000000000010097   CONTEXT: hypervisor
(XEN) rax: 0000000000000002   rbx: 00000000000000ff   rcx: 000000000000038f
(XEN) rdx: 0000000000000000   rsi: 0000000000000004   rdi: 0000000000000100
(XEN) rbp: ffff82d04043c580   rsp: ffff82d0403e7d78   r8:  0000000000000016
(XEN) r9:  0000000000000004   r10: 0000000000000100   r11: 0000000000000000
(XEN) r12: 000000000000038f   r13: ffff82d0403e7e48   r14: 0000000000000001
(XEN) r15: ffff830000095f50   cr0: 0000000080050033   cr4: 00000000000000a0
(XEN) cr3: 00000000bf847000   cr2: 0000000000000000
(XEN) fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen code around <ffff82d040281311> (intel.c#init_intel+0x51/0x670):
(XEN)  bc 8f 03 00 00 44 89 e1 <0f> 32 48 c1 e2 20 89 f1 48 09 c2 b8 01 00 00 00
(XEN) Xen stack trace from rsp=ffff82d0403e7d78:
(XEN)    ffff82d0403bbb69 0000000000000240 ffff82d04043c580 ffff82d04043c58c
(XEN)    ffff82d0403e7e48 0000000000000001 ffff82d0402805e7 0000000000000002
(XEN)    ffff82d04043c700 0000000000000001 ffff82d0403d35ea ffff830000095fb0
(XEN)    ffff830000095f50 ffff830000000002 0000000000000000 ffff830224468000
(XEN)    00000000045e4000 0000000000240000 ffff82d0403e7ef8 ffff830000095f80
(XEN)    00000000000001ff 000000022447b000 0000000000095f80 ffff830000095f50
(XEN)    0000000000000000 0000000000000000 0000000000000002 ffff82d04048c000
(XEN)    ffff82d04058c000 0000000800000000 000000010000006e 0000000000000003
(XEN)    00000000000002f8 2d00000000000000 00095f70bf7b7fde bf7b7fd9bf7b8029
(XEN)    bf7b7b5d00000063 00000004bf7b8038 bf7b803801000000 bf7b88ea00095f60
(XEN)    bf7b7cfe00000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff82d0402757f4
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000e01000000000 0000000000000000 0000000000000000
(XEN)    00000000000000a0 0000000000000000 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN)    [<ffff82d040281311>] R intel.c#init_intel+0x51/0x670
(XEN)    [<ffff82d0403bbb69>] S intel.c#intel_init_levelling+0x9/0x320
(XEN)    [<ffff82d0402805e7>] S identify_cpu+0x287/0x490
(XEN)    [<ffff82d0403d35ea>] S __start_xen+0x14da/0x264a
(XEN)    [<ffff82d0402757f4>] S __high_start+0x94/0xa0
(XEN) 
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) GENERAL PROTECTION FAULT
(XEN) [error_code=0000]
(XEN) ****************************************
(XEN) 
(XEN) Reboot in five seconds...
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.

May 15 '25 18:05 ohault

Hi @marmarek, can you please fix it on your side for QubesOS 4.2.x users, because it has still not been fixed in XEN 4.17 maintenance releases; meanwhile, the severity of this regression is so high that it is landing on an unbootable system?

Here is the original patch - https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=dd05d265b8abda4cc7206b29cd71b77fb46658bf merged in XEN 4.19.2 used to fix this bug it in the XEN 4.19 version.

AFAIK, for AlpineLinux (https://gitlab.alpinelinux.org/alpine/aports/-/issues/17108) and XCP-ng 8.3 (https://github.com/xcp-ng/xen/commit/6bdb965178bbb3fc50cd4418d4770a7789956e2c), a local patch has been already included.

Aug 13 '25 19:08 ohault