QEMU aborted when doing reboot inside the VM

[changpe1]

Environment:

QEMU: vfio-user-v0.6 libvfiouser: latest master with commit ba6abc4ce1

Got the following errors when doing reboot inside VM:

[email protected]:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xbfffffff [0x7fcaa7ec0000]
qemu-system-x86_64: kvm_set_user_memory_region: KVM_SET_USER_MEMORY_REGION failed, slot=4, start=0xc0000, size=0xbff40000: File exists
kvm_set_phys_mem: error registering slot: File exists

[changpe1]

Here the vfio trace in the QEMU side:

[email protected]:vfio_listener_region_add_ram region_add [ram] 0x0 - 0x9ffff [0x7f50b3e00000]
[email protected]:vfio_listener_region_add_skip SKIPPING region_add 0xa0000 - 0xbffff
[email protected]:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xdffff [0x7f50b3200000]
[email protected]:vfio_listener_region_add_ram region_add [ram] 0xe0000 - 0xfffff [0x7f50b3420000]
[email protected]:vfio_listener_region_add_ram region_add [ram] 0x100000 - 0xbfffffff [0x7f50b3f00000]
[email protected]:vfio_listener_region_add_skip SKIPPING region_add 0xfec00000 - 0xfec00fff
[email protected]:vfio_listener_region_add_skip SKIPPING region_add 0xfed00000 - 0xfed003ff
[email protected]:vfio_listener_region_add_skip SKIPPING region_add 0xfee00000 - 0xfeefffff
[email protected]:vfio_listener_region_add_ram region_add [ram] 0xfffc0000 - 0xffffffff [0x7f50b3400000]
[email protected]:vfio_listener_region_add_ram region_add [ram] 0x100000000 - 0x13fffffff [0x7f5173e00000]
[email protected]:vfio_region_sparse_mmap_header Device VFIO user </var/run/muser/domain/muser0/8/cntrl> region 0: 1 sparse mmap entries
[email protected]:vfio_region_sparse_mmap_entry sparse entry 0 [0x1000 - 0x2000]
[email protected]:vfio_region_setup Device VFIO user </var/run/muser/domain/muser0/8/cntrl>, region 0 "VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 0", flags: 0xf, offset: 0x0, size: 0x4000
[email protected]:vfio_region_setup Device VFIO user </var/run/muser/domain/muser0/8/cntrl>, region 1 "VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 1", flags: 0x0, offset: 0x10000000000, size: 0x0
[email protected]:vfio_region_setup Device VFIO user </var/run/muser/domain/muser0/8/cntrl>, region 2 "VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 2", flags: 0x0, offset: 0x20000000000, size: 0x0
[email protected]:vfio_region_setup Device VFIO user </var/run/muser/domain/muser0/8/cntrl>, region 3 "VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 3", flags: 0x0, offset: 0x30000000000, size: 0x0
[email protected]:vfio_region_setup Device VFIO user </var/run/muser/domain/muser0/8/cntrl>, region 4 "VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 4", flags: 0x3, offset: 0x40000000000, size: 0x1000
[email protected]:vfio_region_setup Device VFIO user </var/run/muser/domain/muser0/8/cntrl>, region 5 "VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 5", flags: 0x3, offset: 0x50000000000, size: 0x1000
[email protected]:vfio_populate_device_config Device VFIO user </var/run/muser/domain/muser0/8/cntrl> config:
  size: 0x1000, offset: 0x70000000000, flags: 0x3
[email protected]:vfio_msix_early_setup VFIO user </var/run/muser/domain/muser0/8/cntrl> PCI MSI-X CAP @0x84, BAR 4, offset 0x0, entries 64
[email protected]:vfio_region_mmap Region VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 0 mmaps[0] [0x1000 - 0x1fff]
[email protected]:vfio_intx_enable_kvm  (VFIO user </var/run/muser/domain/muser0/8/cntrl>) KVM INTx accel enabled
[email protected]:vfio_intx_enable  (VFIO user </var/run/muser/domain/muser0/8/cntrl>)
[email protected]:vfio_get_dev_region VFIO user </var/run/muser/domain/muser0/8/cntrl> index 9, 00000003/18
[email protected]:vfio_region_setup Device VFIO user </var/run/muser/domain/muser0/8/cntrl>, region 9 "migration", flags: 0x8, offset: 0x90000000000, size: 0x0
qemu-system-x86_64: -device vfio-user-pci,socket=/var/run/muser/domain/muser0/8/cntrl: VFIO user </var/run/muser/domain/muser0/8/cntrl>: Invalid zero-sized VFIO migration region 9
[email protected]:vfio_intx_disable_kvm  (VFIO user </var/run/muser/domain/muser0/8/cntrl>) KVM INTx accel disabled
[email protected]:vfio_region_mmaps_set_enabled Region VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 0 mmaps enabled: 1
[email protected]:vfio_region_mmaps_set_enabled Region VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 4 mmaps enabled: 1
[email protected]:vfio_region_mmaps_set_enabled Region VFIO user </var/run/muser/domain/muser0/8/cntrl> BAR 5 mmaps enabled: 1
[email protected]:vfio_intx_disable  (VFIO user </var/run/muser/domain/muser0/8/cntrl>)
[email protected]:vfio_pci_read_config  (VFIO user </var/run/muser/domain/muser0/8/cntrl>, @0x44, len=0x2) 0x8
[email protected]:vfio_pci_read_config  (VFIO user </var/run/muser/domain/muser0/8/cntrl>, @0x4, len=0x2) 0x0
[email protected]:vfio_pci_write_config  (VFIO user </var/run/muser/domain/muser0/8/cntrl>, @0x4, 0x0, len=0x2)
[email protected]:vfio_intx_enable  (VFIO user </var/run/muser/domain/muser0/8/cntrl>)
[email protected]:vfio_listener_region_del region_del 0xc0000 - 0xdffff
[email protected]:vfio_listener_region_del region_del 0xe0000 - 0xfffff
[email protected]:vfio_listener_region_del region_del 0x100000 - 0xbfffffff
[email protected]:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xcffff [0x7f50b3ec0000]
[email protected]:vfio_listener_region_add_ram region_add [ram] 0xd0000 - 0xdffff [0x7f50b3210000]
[email protected]:vfio_listener_region_add_ram region_add [ram] 0xe0000 - 0xeffff [0x7f50b3420000]
[email protected]:vfio_listener_region_add_ram region_add [ram] 0xf0000 - 0xbfffffff [0x7f50b3ef0000]
[email protected]:vfio_listener_region_del region_del 0xc0000 - 0xcffff
[email protected]:vfio_listener_region_del region_del 0xc0000 - 0xcffff
[email protected]:vfio_listener_region_del region_del 0xd0000 - 0xdffff
qemu-system-x86_64: vfio_dma_unmap(0x55b8b810c000, 0xc0000, 0x10000) = -2 (Success)
[email protected]:vfio_listener_region_del region_del 0xd0000 - 0xdffff
[email protected]:vfio_listener_region_del region_del 0xe0000 - 0xeffff
qemu-system-x86_64: vfio_dma_unmap(0x55b8b810c000, 0xd0000, 0x10000) = -2 (Success)
[email protected]:vfio_listener_region_del region_del 0xe0000 - 0xeffff
[email protected]:vfio_listener_region_del region_del 0xf0000 - 0xbfffffff
qemu-system-x86_64: vfio_dma_unmap(0x55b8b810c000, 0xe0000, 0x10000) = -2 (Success)
[email protected]:vfio_listener_region_del region_del 0xf0000 - 0xbfffffff
[email protected]:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xbfffffff [0x7f50b3ec0000]
qemu-system-x86_64: vfio_dma_unmap(0x55b8b810c000, 0xf0000, 0xbff10000) = -2 (Success)
[email protected]:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xbfffffff [0x7f50b3ec0000]
qemu-system-x86_64: kvm_set_user_memory_region: KVM_SET_USER_MEMORY_REGION failed, slot=4, start=0xc0000, size=0xbff40000: File exists
kvm_set_phys_mem: error registering slot: File exists

[tmakatos]

I can't reproduce this in my nested setup (KVM is enabled):

qemu-system-x86_64 -cpu host -smp 1 -enable-kvm -nographic -m 2G -object memory-backend-file,id=mem0,size=2G,mem-path=/dev/hugepages,share=on,prealloc=yes, -numa node,memdev=mem0 -kernel bionic-server-cloudimg-amd64-vmlinuz-generic -initrd bionic-server-cloudimg-amd64-initrd-generic -append console=ttyS0 root=/dev/sda1 single -hda bionic-server-cloudimg-amd64-0.raw -device vfio-user-pci,socket=/var/run/muser/domain/muser0/8/cntrl -trace enable=vfio*

This might be a KVM bug. What's your kernel version? I'm testing with 5.4.

[changpe1]

@tmakatos I found the issue can only happen when I used the physical NVMe SSDs as the backend, the difference is SPDK will call spdk_mem_register() to register VM's memory regions to the kernel vfio driver(if the physical NVMe SSD is attached to vfio-pci driver).

For your test case, it should work well.

[changpe1]

I can confirm that when the issue happened, there are still memory region hold by kernel vfio-pci driver(should be unregistered), there maybe mismatch between QEMU/libvfio-user/SPDK, I can still not confirm in which part.

[tmakatos]

I see, looks like I need a very specific setup which will take me some time to prepare. Does this repro reliably? Also, does the passed through disk to SPDK have to be NVMe? Does it still repro with a SCSI disk?

Also, can you provide the SPDK and QEMU commits you're using?

[changpe1]

Yeah, it's very easy to reproduce, even without rebooting inside VM, I can still reproduce it.

1. Start QEMU
2. Kill QEMU
3. Restart QEMU

[changpe1]

[email protected]:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xbfffffff [0x7f50b3ec0000]
qemu-system-x86_64: vfio_dma_unmap(0x55b8b810c000, 0xf0000, 0xbff10000) = -2 (Success)
[email protected]:vfio_listener_region_add_ram region_add [ram] 0xc0000 - 0xbfffffff [0x7f50b3ec0000]

it's not quite clear for me that why the same memory region was added twice, that's the reason why KVM failed with EEXIST.

[changpe1]

For a test comparison, I started VMs with "-device vfio-pci,sysfsdev=/sys/bus/pci/devices/0000:d8:00.0" and "-device vfio-user-pci,socket=/var/run/muser/domain/muser0/8/cntrl", the test with QEMU vfio-pci driver worked well for me, so I can narrow down this issue should related with libvfio-user or SPDK, looks like there are some actions are missed.

Is it related with PCI_RESET? I didn't add that function in SPDK yet.

[tmakatos]

Thanks for providing more information, the reset issue might be related. I'm looking at fixing the migration bug(s) now, can this wait for a bit?

[tmakatos]

@changpe1 with https://github.com/nutanix/libvfio-user/pull/464 fixed, can you try reproing with latest libvfio-user?

[changpe1]

Tested and it passed, closing it for now.

[tmakatos]

Re-opening as I've just hit this bug. This bug triggers on the destination host when trying to migrate a guest:

qemu-system-x86_64: kvm_set_user_memory_region: KVM_SET_USER_MEMORY_REGION failed, slot=10, start=0xfebd1000, size=0x1000: File exists

VFIO trace:

vfio_listener_region_add_ram region_add [ram] 0xfebd1000 - 0xfebd1fff [0x7fa916d1c000]
vfio_listener_region_add_skip SKIPPING region_add 0xfebd0000 - 0xfebd0fff
vfio_listener_region_add_ram region_add [ram] 0xfebd1000 - 0xfebd1fff [0x7fa916d1c000]

This isn't 100% reproducible. Reloading kvm_intel.ko seems to fix it for while. Removing -cpu host -enable-kvm also fixes it. I think that this is something that the mp-qemu folks should look at: https://github.com/oracle/qemu/issues/9