Kernel panic while running on qemu-kvm with latest HEAD 3921

[ader1990]

While testing Flatcar, one of my QEMU-KVM VMs got a kernel panic, posting the errors from the console, in case it might reproduce in the future.

Host OS: Ubuntu Noble 24.04, kernel 6.8.0-11-generic, Intel NUC Extreme 13 box. Host Ubuntu kernel:

uname -a
Linux flatcar-test-box 6.8.0-11-generic #11-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 14 00:29:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

VM was started using the: sudo bash ./flatcar_production_qemu.sh -nographic Flatcar VM kernel:

uname -a
Linux localhost 6.6.21-flatcar #1 SMP PREEMPT_DYNAMIC Wed Mar 27 17:21:51 -00 2024 x86_64 13th Gen Intel(R) Core(TM) i5-13600K GenuineIntel GNU/Linux

Unknown cause or repro yet.

[59451.222618] NMI backtrace for cpu 12
[59451.222624] CPU: 12 PID: 0 Comm: swapper/12 Tainted: G             L     6.6.21-flatcar #1
[59451.222625] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.0-20220807_005459-localhost 04/01/2014
[59451.222626] RIP: 0010:delay_halt_tpause+0x1a/0x20
[59451.222631] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 0f 1f 44 00 00 48 8d 04 37 31 c9 48 89 c2 48 c1 ea 20 66 0f ae f1 <c3> cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90
[59451.222632] RSP: 0018:ffffb7d64031caa0 EFLAGS: 00000002
[59451.222633] RAX: 0000bcf45eb7bcc3 RBX: 0000000000000da7 RCX: 0000000000000000
[59451.222634] RDX: 000000000000bcf4 RSI: 0000000000000da7 RDI: 0000bcf45eb7af1c
[59451.222634] RBP: 0000bcf45eb7af1c R08: ffffffffa62688a0 R09: 00000000a974d0aa
[59451.222635] R10: ffffffffffffffff R11: 0000000000000002 R12: 0000000000000000
[59451.222635] R13: 00000000ee6b2800 R14: ffff8f663ed20d00 R15: ffff8f663ed20580
[59451.222636] FS:  0000000000000000(0000) GS:ffff8f663ed00000(0000) knlGS:0000000000000000
[59451.222637] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[59451.222638] CR2: 00007f6fc8efe580 CR3: 00000000293c0000 CR4: 0000000000750ee0
[59451.222640] PKRU: 55555554
[59451.222641] Call Trace:
[59451.222646]  <NMI>
[59451.222646]  ? nmi_cpu_backtrace+0x99/0x110
[59451.222651]  ? nmi_cpu_backtrace_handler+0x11/0x20
[59451.222653]  ? nmi_handle+0x5e/0x150
[59451.222662]  ? default_do_nmi+0x40/0x100
[59451.222663]  ? exc_nmi+0x102/0x130
[59451.222664]  ? end_repeat_nmi+0x16/0x67
[59451.222666]  ? delay_halt_tpause+0x1a/0x20
[59451.222668]  ? delay_halt_tpause+0x1a/0x20
[59451.222669]  ? delay_halt_tpause+0x1a/0x20
[59451.222671]  </NMI>
[59451.222671]  <IRQ>
[59451.222672]  delay_halt+0x3c/0x70
[59451.222674]  native_stop_other_cpus+0x1d4/0x270
[59451.222677]  panic+0x187/0x320
[59451.222679]  watchdog_timer_fn+0x1e2/0x220
[59451.222681]  ? __pfx_watchdog_timer_fn+0x10/0x10
[59451.222683]  __hrtimer_run_queues+0x10f/0x2b0
[59451.222685]  hrtimer_interrupt+0xf8/0x230
[59451.222686]  __sysvec_apic_timer_interrupt+0x4d/0x140
[59451.222688]  sysvec_apic_timer_interrupt+0x39/0x90
[59451.222689]  asm_sysvec_apic_timer_interrupt+0x1a/0x20
[59451.222691] RIP: 0010:console_flush_all+0x1fa/0x4e0
[59451.222692] Code: 74 24 04 48 c7 c7 80 89 34 a6 e8 e1 c7 01 00 45 88 3c 24 e8 58 28 00 00 f7 c5 00 02 00 00 0f 84 d4 00 00 00 fb 0f 1f 44 00 00 <41> 80 3c 24 00 0f 85 bf 00 00 00 49 8b 46 58 49 39 45 00 73 04 49
[59451.222693] RSP: 0018:ffffb7d64031cd18 EFLAGS: 00000206
[59451.222693] RAX: 000000000001d6c0 RBX: 0000000000000001 RCX: 00000000ffffffff
[59451.222694] RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffffffa6348980
[59451.222694] RBP: 0000000000000246 R08: 0000000000000000 R09: 6461657220676e6f
[59451.222695] R10: 0000000000000031 R11: 00000000636f6c63 R12: ffffb7d64031cdaf
[59451.222696] R13: ffffb7d64031cdb0 R14: ffffffffa63c9180 R15: 0000000000000001
[59451.222699]  ? console_flush_all+0x1e8/0x4e0
[59451.222700]  console_unlock+0x4a/0x110
[59451.222701]  vprintk_emit+0x149/0x2a0
[59451.222702]  _printk+0x5c/0x80
[59451.222704]  clocksource_watchdog+0x5cc/0x600
[59451.222706]  ? __pfx_clocksource_watchdog+0x10/0x10
[59451.222708]  ? __pfx_clocksource_watchdog+0x10/0x10
[59451.222710]  call_timer_fn+0x24/0x130
[59451.222712]  __run_timers.part.0+0x1d8/0x280
[59451.222714]  ? __hrtimer_run_queues+0x121/0x2b0
[59451.222714]  ? kvm_clock_get_cycles+0x18/0x30
[59451.222716]  ? ktime_get+0x39/0xa0
[59451.222719]  ? kvm_sched_clock_read+0x11/0x20
[59451.222720]  ? sched_clock+0x10/0x30
[59451.222722]  run_timer_softirq+0x2a/0x50
[59451.222724]  __do_softirq+0xd1/0x2bf
[59451.222725]  __irq_exit_rcu+0x76/0xa0
[59451.222726]  sysvec_apic_timer_interrupt+0x72/0x90
[59451.222727]  </IRQ>
[59451.222728]  <TASK>
[59451.222728]  asm_sysvec_apic_timer_interrupt+0x1a/0x20
[59451.222729] RIP: 0010:pv_native_safe_halt+0xf/0x20
[59451.222731] Code: 0b 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa eb 07 0f 00 2d 25 b7 23 00 fb f4 <c3> cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 90 90 90
[59451.222731] RSP: 0018:ffffb7d6400fbed8 EFLAGS: 00000206
[59451.222732] RAX: ffff8f663ed2b220 RBX: 000000000000000c RCX: 4000000000000000
[59451.222732] RDX: 0000000000000001 RSI: ffffffffa5b9aeae RDI: 0000000000c18534
[59451.222733] RBP: ffff8f6601352080 R08: 0000000000c18534 R09: 0000000000000000
[59451.222733] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
[59451.222734] R13: 0000000000000000 R14: ffff8f6601352080 R15: 0000000000000000
[59451.222735]  ? ct_kernel_exit.constprop.0+0x71/0xe0
[59451.222737]  default_idle+0x9/0x20
[59451.222738]  default_idle_call+0x2c/0xe0
[59451.222740]  do_idle+0x1ec/0x230
[59451.222742]  cpu_startup_entry+0x2a/0x30
[59451.222743]  start_secondary+0x11e/0x140
[59451.222745]  secondary_startup_64_no_verify+0x18a/0x18b
[59451.222747]  </TASK>
[59452.272973] Shutting down cpus with NMI
[59452.373922] Kernel Offset: 0x23000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[59452.374590] Rebooting in 60 seconds.