oak
oak copied to clipboard
project-oak
Oak Containers: QEMU not recognizing SEV-SNP: `parameter 'qom-type' does not accept value 'sev-snp-guest'`
I'm working on the main branch (commit c5190c07aa8da7436760f39bb5de8e27afb44844).
I'm attempting to run the rust_hello_world_trusted_bundle container using the following command:
sudo RUST_LOG=debug ./target/x86_64-unknown-linux-gnu/release/oak_containers_launcher \
--system-image oak_containers/system_image/target/image.tar.xz \
--container-bundle target/rust_hello_world_trusted_bundle.tar \
--vmm-binary $(which qemu-system-x86_64) \
--stage0-binary generated/stage0_bin \
--kernel oak_containers/kernel/target/bzImage \
--initrd target/stage1.cpio \
--ramdrive-size 5000000 \
--vm-type sev-snp
Running this command results in the following log lines:
[2024-10-11T20:18:40Z INFO oak_containers_launcher] Launcher service listening on port 43977
[2024-10-11T20:18:40Z DEBUG oak_containers_launcher::qemu] QEMU command line: Command { std: "/nix/store/6x9r5ghwq2a6j4r2gmqida8d5wd0ya91-qemu-host-cpu-only-9.0.1/bin/qemu-system-x86_64" "-enable-kvm" "-cpu" "host" "-smp" "1" "-nodefaults" "-nographic" "-no-reboot" "-machine" "microvm,acpi=on,pcie=on,confidential-guest-support=sev0,memory-backend=ram1" "-object" "memory-backend-memfd,id=ram1,size=8G,share=true,reserve=false" "-object" "sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,id-auth=" "-chardev" "socket,id=consock,fd=11" "-serial" "chardev:consock" "-netdev" "user,id=netdev,guestfwd=tcp:10.0.2.100:8080-cmd:nc 127.0.0.1 43977,hostfwd=tcp:127.0.0.1:41161-10.0.2.15:4000,hostfwd=tcp:127.0.0.1:45071-10.0.2.15:8080" "-device" "virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=netdev,romfile=" "-device" "vhost-vsock-pci,guest-cid=1748300,rombar=0" "-bios" "generated/stage0_bin" "-kernel" "oak_containers/kernel/target/bzImage" "-initrd" "target/stage1.cpio" "-append" " console=ttyS0 panic=-1 brd.rd_nr=1 brd.rd_size=5000000 brd.max_part=1 ip=10.0.2.15:::255.255.255.0::eth0:off loglevel=7 -- --launcher-addr=vsock://2:43977", kill_on_drop: true }
qemu-system-x86_64: -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,id-auth=: Parameter 'qom-type' does not accept value 'sev-snp-guest'
The last log line indicates a QEMU issue.
Our setup is:
- server: Supermicro CloudDC A+ Server AS -2015CS-TNR
- cpu: 1x EPYC 7354P 32C/64T CPU. This is an EPYC 4th Generation AMD Genoa chip, and supports SEV-SNP.
- kernel: 6.11.0-rc3-snp-host-85ef1ac03941. We followed AMD's Using SEV with AMD EPYC Processors guide to setup the BIOS, SEV firmware, and host kernel. Specifically, we're running AMD's kernel as per the instructions on AMD's sep-latest branch
- nix: We're using the nix-2.14.1 setup as described in
docs/development.md.
Additionally, I manually had to sudo modprobe vhost_vsock.
Thanks. I appreciated any help.
I believe that AMD SEV-SNP support has not been upstreamed for QEMU yet, so the version of QEMU that is installed in the nix environment will not work with it. To use SEV-SNP I think you will have to install the version from the AMD repository and use that version in the launcher.
To further clarify, we are using the nix install of qemu:
$ which qemu-system-x86_64
/nix/store/6x9r5ghwq2a6j4r2gmqida8d5wd0ya91-qemu-host-cpu-only-9.0.1/bin/qemu-system-x86_64
$ qemu-system-x86_64 --version
QEMU emulator version 9.0.1
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers
I think that you will have to build QEMU from https://github.com/AMDESE/qemu/tree/snp-latest to support AMD SEV-SNP rather than use the nix version
@conradgrobler: thank you; that worked after applying @dingelish's QEMU patch dingelish/qemu@876e262 to also correct the vmw_vsock_virtio_transport error at launch.
@conradgrobler we verified the stack on the latest upstream stable 6.11 kernel and upstream qemu. AMD's branch is bit of old.
@smherwig could you please do me a favor? I want to know if you can run the vanilla qemu without that patch after applying this to the oak_containers_launcher
diff --git a/oak_containers/launcher/src/qemu.rs b/oak_containers/launcher/src/qemu.rs
index 487d46865..71890ac2d 100644
--- a/oak_containers/launcher/src/qemu.rs
+++ b/oak_containers/launcher/src/qemu.rs
@@ -168,7 +168,7 @@ impl Qemu {
params.memory_size.clone().unwrap_or("8G".to_string())
);
// SEV's feature configuration.
- let sev_config_object = "id=sev0,cbitpos=51,reduced-phys-bits=1";
+ let sev_config_object = "id=sev0,cbitpos=51,reduced-phys-bits=2";
// TDX machine suffix
let tdx_machine_suffix = ",kernel_irqchip=split,memory-encryption=tdx,memory-backend=ram1";
let tdx_common_object = format!(
also this patch
diff --git a/oak_containers/launcher/src/qemu.rs b/oak_containers/launcher/src/qemu.rs
index 487d46865..71890ac2d 100644
--- a/oak_containers/launcher/src/qemu.rs
+++ b/oak_containers/launcher/src/qemu.rs
@@ -168,7 +168,7 @@ impl Qemu {
params.memory_size.clone().unwrap_or("8G".to_string())
);
// SEV's feature configuration.
- let sev_config_object = "id=sev0,cbitpos=51,reduced-phys-bits=1";
+ let sev_config_object = "id=sev0,cbitpos=51";
// TDX machine suffix
let tdx_machine_suffix = ",kernel_irqchip=split,memory-encryption=tdx,memory-backend=ram1";
let tdx_common_object = format!(
@dingelish - Thanks. For either patch, I still get the QEMU error:
Parameter 'qom-type' does not accept value 'sev-snp-guest'
when using vanilla QEMU v9.0.1 (/nix/store/6x9r5ghwq2a6j4r2gmqida8d5wd0ya91-qemu-host-cpu-only-9.0.1/bin/qemu-system-x86_64).
Patch 1 Trial Run
$ sudo RUST_LOG=debug ./target/x86_64-unknown-linux-gnu/release/oak_containers_hello_world_untrusted_app \
--system-image oak_containers/system_image/target/image.tar.xz \
--container-bundle target/rust_hello_world_trusted_bundle.tar \
--vmm-binary $(which qemu-system-x86_64) \
--stage0-binary generated/stage0_bin \
--kernel oak_containers/kernel/target/bzImage \
--initrd target/stage1.cpio \
--ramdrive-size 5000000 \
--vm-type sev-snp rest
ARGS: Args { server_type: Rest, launcher_args: Args { system_image: "oak_containers/system_image/target/image.tar.xz", container_bundle: "target/rust_hello_world_trusted_bundle.tar", application_config: [], qemu_params: Params { vmm_binary: "/nix/store/6x9r5ghwq2a6j4r2gmqida8d5wd0ya91-qemu-host-cpu-only-9.0.1/bin/qemu-system-x86_64", stage0_binary: "generated/stage0_bin", kernel: "oak_containers/kernel/target/bzImage", initrd: "target/stage1.cpio", memory_size: Some("8G"), num_cpus: 1, ramdrive_size: 5000000, telnet_console: None, virtio_guest_cid: None, pci_passthrough: None, vm_type: SevSnp }, communication_channel: Network } }
SERVER ADDR Ok(0.0.0.0:8006)
[2024-10-16T20:06:59Z INFO oak_containers_launcher] Launcher service listening on port 34419
[2024-10-16T20:06:59Z DEBUG oak_containers_launcher::qemu] QEMU command line: Command { std: "/nix/store/6x9r5ghwq2a6j4r2gmqida8d5wd0ya91-qemu-host-cpu-only-9.0.1/bin/qemu-system-x86_64" "-enable-kvm" "-cpu" "host" "-m" "8G" "-smp" "1" "-nodefaults" "-nographic" "-no-reboot" "-machine" "microvm,acpi=on,pcie=on,confidential-guest-support=sev0,memory-backend=ram1" "-object" "memory-backend-memfd,id=ram1,size=8G,share=true,reserve=false" "-object" "sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=2,id-auth=" "-chardev" "socket,id=consock,fd=12" "-serial" "chardev:consock" "-netdev" "user,id=netdev,guestfwd=tcp:10.0.2.100:8080-cmd:nc 127.0.0.1 34419,hostfwd=tcp:127.0.0.1:37327-10.0.2.15:4000,hostfwd=tcp:127.0.0.1:34865-10.0.2.15:8080" "-device" "virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=netdev,romfile=" "-device" "vhost-vsock-pci,guest-cid=145077,rombar=0" "-bios" "generated/stage0_bin" "-kernel" "oak_containers/kernel/target/bzImage" "-initrd" "target/stage1.cpio" "-append" " console=ttyS0 panic=-1 brd.rd_nr=1 brd.rd_size=5000000 brd.max_part=1 ip=10.0.2.15:::255.255.255.0::eth0:off loglevel=7 -- --launcher-addr=vsock://2:34419", kill_on_drop: true }
qemu-system-x86_64: -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=2,id-auth=: Parameter 'qom-type' does not accept value 'sev-snp-guest'
Patch 2 Trial Run
$ sudo RUST_LOG=debug ./target/x86_64-unknown-linux-gnu/release/oak_containers_hello_world_untrusted_app \
--system-image oak_containers/system_image/target/image.tar.xz \
--container-bundle target/rust_hello_world_trusted_bundle.tar \
--vmm-binary $(which qemu-system-x86_64) \
--stage0-binary generated/stage0_bin \
--kernel oak_containers/kernel/target/bzImage \
--initrd target/stage1.cpio \
--ramdrive-size 5000000 \
--vm-type sev-snp rest
ARGS: Args { server_type: Rest, launcher_args: Args { system_image: "oak_containers/system_image/target/image.tar.xz", container_bundle: "target/rust_hello_world_trusted_bundle.tar", application_config: [], qemu_params: Params { vmm_binary: "/nix/store/6x9r5ghwq2a6j4r2gmqida8d5wd0ya91-qemu-host-cpu-only-9.0.1/bin/qemu-system-x86_64", stage0_binary: "generated/stage0_bin", kernel: "oak_containers/kernel/target/bzImage", initrd: "target/stage1.cpio", memory_size: Some("8G"), num_cpus: 1, ramdrive_size: 5000000, telnet_console: None, virtio_guest_cid: None, pci_passthrough: None, vm_type: SevSnp }, communication_channel: Network } }
SERVER ADDR Ok(0.0.0.0:8006)
[2024-10-16T20:11:37Z INFO oak_containers_launcher] Launcher service listening on port 37075
[2024-10-16T20:11:37Z DEBUG oak_containers_launcher::qemu] QEMU command line: Command { std: "/nix/store/6x9r5ghwq2a6j4r2gmqida8d5wd0ya91-qemu-host-cpu-only-9.0.1/bin/qemu-system-x86_64" "-enable-kvm" "-cpu" "host" "-m" "8G" "-smp" "1" "-nodefaults" "-nographic" "-no-reboot" "-machine" "microvm,acpi=on,pcie=on,confidential-guest-support=sev0,memory-backend=ram1" "-object" "memory-backend-memfd,id=ram1,size=8G,share=true,reserve=false" "-object" "sev-snp-guest,id=sev0,cbitpos=51,id-auth=" "-chardev" "socket,id=consock,fd=12" "-serial" "chardev:consock" "-netdev" "user,id=netdev,guestfwd=tcp:10.0.2.100:8080-cmd:nc 127.0.0.1 37075,hostfwd=tcp:127.0.0.1:41391-10.0.2.15:4000,hostfwd=tcp:127.0.0.1:42761-10.0.2.15:8080" "-device" "virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=netdev,romfile=" "-device" "vhost-vsock-pci,guest-cid=149753,rombar=0" "-bios" "generated/stage0_bin" "-kernel" "oak_containers/kernel/target/bzImage" "-initrd" "target/stage1.cpio" "-append" " console=ttyS0 panic=-1 brd.rd_nr=1 brd.rd_size=5000000 brd.max_part=1 ip=10.0.2.15:::255.255.255.0::eth0:off loglevel=7 -- --launcher-addr=vsock://2:37075", kill_on_drop: true }
qemu-system-x86_64: -object sev-snp-guest,id=sev0,cbitpos=51,id-auth=: Parameter 'qom-type' does not accept value 'sev-snp-guest'
@dingelish - It also occurred to me that by "vanilla" you meant the https://github.com/AMDESE/qemu/tree/snp-latest version (without your QEMU patch dingelish/qemu@876e262).
When I try using that version, I get the following errors:
Patch 1 Trial Run
$ sudo RUST_LOG=debug ./target/x86_64-unknown-linux-gnu/release/oak_containers_hello_world_untrusted_app \
--system-image oak_containers/system_image/target/image.tar.xz \
--container-bundle target/rust_hello_world_trusted_bundle.tar \
--vmm-binary /home/smherwig/toil/machine/AMDSEV/snp-release-2024-10-01/usr/local/bin/qemu-system-x86_64 \
--stage0-binary generated/stage0_bin \
--kernel oak_containers/kernel/target/bzImage \
--initrd target/stage1.cpio \
--ramdrive-size 5000000 \
--vm-type sev-snp rest
ARGS: Args { server_type: Rest, launcher_args: Args { system_image: "oak_containers/system_image/target/image.tar.xz", container_bundle: "target/rust_hello_world_trusted_bundle.tar", application_config: [], qemu_params: Params { vmm_binary: "/home/smherwig/toil/machine/AMDSEV/snp-release-2024-10-01/usr/local/bin/qemu-system-x86_64", stage0_binary: "generated/stage0_bin", kernel: "oak_containers/kernel/target/bzImage", initrd: "target/stage1.cpio", memory_size: Some("8G"), num_cpus: 1, ramdrive_size: 5000000, telnet_console: None, virtio_guest_cid: None, pci_passthrough: None, vm_type: SevSnp }, communication_channel: Network } }
SERVER ADDR Ok(0.0.0.0:8006)
[2024-10-16T20:22:59Z INFO oak_containers_launcher] Launcher service listening on port 33725
[2024-10-16T20:22:59Z DEBUG oak_containers_launcher::qemu] QEMU command line: Command { std: "/home/smherwig/toil/machine/AMDSEV/snp-release-2024-10-01/usr/local/bin/qemu-system-x86_64" "-enable-kvm" "-cpu" "host" "-m" "8G" "-smp" "1" "-nodefaults" "-nographic" "-no-reboot" "-machine" "microvm,acpi=on,pcie=on,confidential-guest-support=sev0,memory-backend=ram1" "-object" "memory-backend-memfd,id=ram1,size=8G,share=true,reserve=false" "-object" "sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=2,id-auth=" "-chardev" "socket,id=consock,fd=12" "-serial" "chardev:consock" "-netdev" "user,id=netdev,guestfwd=tcp:10.0.2.100:8080-cmd:nc 127.0.0.1 33725,hostfwd=tcp:127.0.0.1:39829-10.0.2.15:4000,hostfwd=tcp:127.0.0.1:40269-10.0.2.15:8080" "-device" "virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=netdev,romfile=" "-device" "vhost-vsock-pci,guest-cid=155995,rombar=0" "-bios" "generated/stage0_bin" "-kernel" "oak_containers/kernel/target/bzImage" "-initrd" "target/stage1.cpio" "-append" " console=ttyS0 panic=-1 brd.rd_nr=1 brd.rd_size=5000000 brd.max_part=1 ip=10.0.2.15:::255.255.255.0::eth0:off loglevel=7 -- --launcher-addr=vsock://2:33725", kill_on_drop: true }
stage0 INFO: starting...
stage0 DEBUG: Using fw_cfg to create the E820 table
stage0 DEBUG: early E820 entry: [0x0000000000000000-0x0000000000080000), len 524288, type RAM
stage0 DEBUG: early E820 entry: [0x0000000000080000-0x00000000000a0000), len 131072, type ACPI
stage0 DEBUG: early E820 entry: [0x00000000000f0000-0x0000000000100000), len 65536, type RESERVED
stage0 DEBUG: early E820 entry: [0x0000000000100000-0x00000000c0000000), len 3220176896, type RAM
stage0 DEBUG: early E820 entry: [0x00000000feffc000-0x00000000ff000000), len 16384, type RESERVED
stage0 DEBUG: early E820 entry: [0x0000000100000000-0x0000000240000000), len 5368709120, type RAM
stage0 INFO: Enabled SEV features: SevStatus(SEV_ENABLED | SEV_ES_ENABLED | SNP_ACTIVE)
stage0 INFO: starting SEV-SNP memory validation
stage0 INFO: SEV-SNP memory validation complete.
stage0 INFO: Validated using 2 MiB pages: 0
stage0 INFO: Validated using 4 KiB pages: 2096912
stage0 INFO: Valid state not updated: 0
stage0 INFO: RMP page size mismatch errors (fallback to 4K): 4095
stage0 DEBUG: Kernel cmdline: console=ttyS0 panic=-1 brd.rd_nr=1 brd.rd_size=5000000 brd.max_part=1 ip=10.0.2.15:::255.255.255.0::eth0:off loglevel=7 -- --launcher-addr=vsock://2:33725
stage0 DEBUG: Kernel image size 5726208
stage0 DEBUG: Kernel image start address 0x0000000002000000
stage0 DEBUG: Kernel entry point 0x0000000002000200
stage0 INFO: Using x2APIC for AP initialization.
stage0 DEBUG: Local APIC: ProcessorLocalApic { header: ControllerHeader { structure_type: 0, len: 8 }, processor_uid: 0, apic_id: 0, flags: LocalApicFlags(ENABLED) }
stage0 DEBUG: uninteresting structure: 1
stage0 DEBUG: uninteresting structure: 1
stage0 DEBUG: uninteresting structure: 4
stage0 INFO: Expected number of APs: 0, started number of APs: 0
stage0 DEBUG: Initial RAM disk size 2802132
stage0 DEBUG: Initial RAM disk address 0x000000003fd53000
stage0 ERROR: panicked at stage0/src/paging.rs:254:5:
assertion failed: page_start < Size2MiB::SIZE
^C
Patch 2 Trial Run
$ sudo RUST_LOG=debug ./target/x86_64-unknown-linux-gnu/release/oak_containers_hello_world_untrusted_app \
--system-image oak_containers/system_image/target/image.tar.xz \
--container-bundle target/rust_hello_world_trusted_bundle.tar \
--vmm-binary /home/smherwig/toil/machine/AMDSEV/snp-release-2024-10-01/usr/local/bin/qemu-system-x86_64 \
--stage0-binary generated/stage0_bin \
--kernel oak_containers/kernel/target/bzImage \
--initrd target/stage1.cpio \
--ramdrive-size 5000000 \
--vm-type sev-snp rest
ARGS: Args { server_type: Rest, launcher_args: Args { system_image: "oak_containers/system_image/target/image.tar.xz", container_bundle: "target/rust_hello_world_trusted_bundle.tar", application_config: [], qemu_params: Params { vmm_binary: "/home/smherwig/toil/machine/AMDSEV/snp-release-2024-10-01/usr/local/bin/qemu-system-x86_64", stage0_binary: "generated/stage0_bin", kernel: "oak_containers/kernel/target/bzImage", initrd: "target/stage1.cpio", memory_size: Some("8G"), num_cpus: 1, ramdrive_size: 5000000, telnet_console: None, virtio_guest_cid: None, pci_passthrough: None, vm_type: SevSnp }, communication_channel: Network } }
SERVER ADDR Ok(0.0.0.0:8006)
[2024-10-16T20:21:05Z INFO oak_containers_launcher] Launcher service listening on port 41409
[2024-10-16T20:21:05Z DEBUG oak_containers_launcher::qemu] QEMU command line: Command { std: "/home/smherwig/toil/machine/AMDSEV/snp-release-2024-10-01/usr/local/bin/qemu-system-x86_64" "-enable-kvm" "-cpu" "host" "-m" "8G" "-smp" "1" "-nodefaults" "-nographic" "-no-reboot" "-machine" "microvm,acpi=on,pcie=on,confidential-guest-support=sev0,memory-backend=ram1" "-object" "memory-backend-memfd,id=ram1,size=8G,share=true,reserve=false" "-object" "sev-snp-guest,id=sev0,cbitpos=51,id-auth=" "-chardev" "socket,id=consock,fd=12" "-serial" "chardev:consock" "-netdev" "user,id=netdev,guestfwd=tcp:10.0.2.100:8080-cmd:nc 127.0.0.1 41409,hostfwd=tcp:127.0.0.1:42245-10.0.2.15:4000,hostfwd=tcp:127.0.0.1:41525-10.0.2.15:8080" "-device" "virtio-net-pci,disable-legacy=on,iommu_platform=true,netdev=netdev,romfile=" "-device" "vhost-vsock-pci,guest-cid=151994,rombar=0" "-bios" "generated/stage0_bin" "-kernel" "oak_containers/kernel/target/bzImage" "-initrd" "target/stage1.cpio" "-append" " console=ttyS0 panic=-1 brd.rd_nr=1 brd.rd_size=5000000 brd.max_part=1 ip=10.0.2.15:::255.255.255.0::eth0:off loglevel=7 -- --launcher-addr=vsock://2:41409", kill_on_drop: true }
qemu-system-x86_64: -object sev-snp-guest,id=sev0,cbitpos=51,id-auth=: Parameter 'reduced-phys-bits' is missing
@smherwig I upgraded my workstation to use the upstream linux kernel from kernel.org, and upstream 9.1.1 qemu. they worked together perfectly without any patch. I'm using debian 12 bookworm on that machine.