How do you test heads with QEMU?

[itay-grudev]

Kyle Ranking told me that when playing with heads one option to test is to simulate it with QEMU and some of you are using it. That will be much more convenient than re-flashing every time, but when I tried setting it up I couldn't.

Do you have a script or can you provide the qemu command with all the args and options you use?

[ThatLurker]

Is this what your looking for https://github.com/osresearch/heads-wiki/blob/master/Emulating-Heads.md

[tlaurion]

@itay-grudev : I think make run does it, but wasn't able to simply test it.

Emulating Heads needs to be updated. Heads doesn't include Xen anymore since kexec got fixed.

[itay-grudev]

I'm compiling the qemu-coreboot image right now. Will give feedback later today.

[tlaurion]

if anyone knows how to play around qemu in docker, that would be aweome to share. I innocently tried here, without success.

[tlaurion]

Another Heads issue directs to bypass coreboot booting and instruct to boot kernel and initrd, which unfortunately misses the point of measuring coreboot.

[tlaurion]

@itay-grudev how were your tests? :)

[itay-grudev]

make run fails with:

❯ make run
qemu-system-x86_64 \
	--machine q35 \
	--serial /dev/tty \
	--bios /home/ito/Projects/heads/build/qemu-coreboot/coreboot.rom \
; stty sane
qemu-system-x86_64: Initialization of device e1000e failed: failed to find romfile "efi-e1000e.rom"

[paulmenzel]

On 02/15/19 11:52, Itay Grudev wrote:

make run fails with:

❯ make run
qemu-system-x86_64 \
	--machine q35 \
	--serial /dev/tty \
	--bios /home/ito/Projects/heads/build/qemu-coreboot/coreboot.rom \
; stty sane
qemu-system-x86_64: Initialization of device e1000e failed: failed to find romfile "efi-e1000e.rom"

As a workaround, please add -nic none to that call, and check, if it fixes the issue. Otherwise, you need to find the option ROM somewhere.

[itay-grudev]

Thanks. I was able to get it working with:

diff --git a/boards/qemu-coreboot/qemu-coreboot.config b/boards/qemu-coreboot/qemu-coreboot.config
index 9427a2a..a0520fe 100644
--- a/boards/qemu-coreboot/qemu-coreboot.config
+++ b/boards/qemu-coreboot/qemu-coreboot.config
@@ -42,4 +42,5 @@ run:
                --machine q35 \
                --serial /dev/tty \
                --bios $(build)/$(BOARD)/coreboot.rom \
+               -nic none \
        ; stty sane

[itay-grudev]

Do you know how can I get the GUI working? I am currently getting:

Error: Can't find usable screen

It's probably some QEMU option, I just can't find it.

[tlaurion]

FYI: debian-9 can install qemu tools in a TemplateVM, fedora-29 can't for the moment. It will be fixed in QubesOS 4.1.

I gave up a long time ago trying to build Heads in debian templates, and CI based on ubuntu are currently failing @osresearch.

[tlaurion]

Do you know how can I get the GUI working? I am currently getting:

Error: Can't find usable screen

It's probably some QEMU option, I just can't find it.

@itay-grudev : https://github.com/flammit/heads/tree/qemu-gui-init Ping me back if that works. That should probably merged in master too.

[itay-grudev]

Yep. That works. I realised that some of the settings were missing, but didn't replicate them exactly. And that PR did it. Thanks a lot. I will close this and vote on your #527 PR.

@tlaurion Should we also create a PR with -nic none (diff above)?

[tlaurion]

@itay-grudev @paulmenzel Are you able to access a block device under qemu-coreboot build? qemu-img create -f qcow2 -o size=10G ubuntu.img

Documentation source

Drive option

qemu-system-x86_64 \
	--machine q35 \
	--serial /dev/tty \
	--bios /home/user/heads/build/qemu-coreboot/coreboot.rom \
        -drive file=/media/user/Insurgo/ubuntu.img

hda standard support:

qemu-system-x86_64 \
	--machine q35 \
	--serial /dev/tty \
	--bios /home/user/heads/build/qemu-coreboot/coreboot.rom \
        -hda /media/user/Insurgo/ubuntu.img

ide-drive: qemu-system-x86_64 --machine q35 --serial /dev/tty --bios /home/user/heads/build/qemu-coreboot/coreboot.rom -device ide-drive,bus=ide.0,drive=testhdd -drive id=testhdd,if=none,file=/media/user/Insurgo/ubuntu.img

AHCI qemu-system-x86_64 --machine q35 -m 1G --serial stdio --bios build/qemu-coreboot/coreboot.rom -drive id=disk,file=/media/user/Insurgo/ubuntu.img,if=none,format=qcow2 -device ahci,id=ahci -device ide-drive,drive=disk,bus=ahci.0 -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0

There is no /dev/sda under Heads (debian-9 QubesOS template with qemu deployed, so no kvm). Only /sys/block/loop* and /sys/block/ram* devices.

Was wondering if I was doing something wrong or if there is a concept i'm missing?

[tlaurion]

@flammit used: qemu-system-x86_64 --enable-kvm --machine q35 -m 1G --serial stdio --vga cirrus --bios build/qemu-coreboot/coreboot.rom -hda ../test.hd1.img --drive id=stick,if=none,format=raw,file=/home/flam/Downloads/Fedora-Workstation-Live-x86_64-29-1.2.iso -usb -device usb-ehci,id=ehci -device usb-tablet,bus=usb-bus.0 -device usb-storage,bus=ehci.0,drive=stick

[tlaurion]

Trying to find my way with qemu without KVM since QubesOS doesn't support nested virtualization.

Works on QubesOS debian-9 template based qube, on which qemu was installed on parent template.

Didn't found a way to pass devices yet, which is problematic, but I can at least test script changes without having to flash real hardware internally for everything.

gui-init requires an existing /boot. My testing requires a functional /media where oem-provisioning is found.

Create disks and required content to be tested:

cd ~/heads
dd if=/dev/zero of=initrd/boot.img bs=1M count=1
sudo mkfs.ext4 initrd/boot.img
sudo mount initrd/boot.img /mnt/
sudo touch /mnt/oem
sudo umount /mnt
dd if=/dev/zero of=initrd/media.img bs=1M count=1
sudo mkfs.ext4 initrd/media.img
sudo mount  initrd/media.img /mnt
sudo cp oem-reownership/oem-provisioning.example /mnt/oem-provisioning

Note: unfortunately, minimal size for LUKS container is 1049600 bytes, and adding a 500Mb + disk image into rom is not desirable. Wish I could find a way to test this by providing qemu-image disks to qemu without hacking my way like this, but it really seems kvm is required for this, even when "if=none" is passed, which from my understanding doesn't depend on virtio internals. Ho well.

Make those disks mounted by /initrd/init which feeds /etc/fstab correctly:

diff --git a/boards/qemu-coreboot/qemu-coreboot.config b/boards/qemu-coreboot/qemu-coreboot.config
index d670dc2..142c7f4 100644
--- a/boards/qemu-coreboot/qemu-coreboot.config
+++ b/boards/qemu-coreboot/qemu-coreboot.config
@@ -24,12 +24,12 @@ CONFIG_DROPBEAR=y
 
 #Uncomment only one of the following block
 #Required for graphical gui-init (FBWhiptail)
-#CONFIG_CAIRO=y
-#CONFIG_FBWHIPTAIL=y
+CONFIG_CAIRO=y
+CONFIG_FBWHIPTAIL=y
 #
 #text-based init (generic-init and gui-init)
-CONFIG_NEWT=y
-CONFIG_SLANG=y
+#CONFIG_NEWT=y
+#CONFIG_SLANG=y
 
 endif
 
@@ -40,15 +40,15 @@ CONFIG_LINUX_E1000=y
 
 #Uncomment only one BOOTSCRIPT:
 #Whiptail-based init (text-based or FBWhiptail)
-#export CONFIG_BOOTSCRIPT=/bin/gui-init
+export CONFIG_BOOTSCRIPT=/bin/gui-init
 #
 #text-based original init:
-export CONFIG_BOOTSCRIPT=/bin/generic-init
+#export CONFIG_BOOTSCRIPT=/bin/generic-init
 
 export CONFIG_TPM=n
 
-export CONFIG_BOOT_DEV="/dev/sda1"
-export CONFIG_USB_BOOT_DEV="/dev/sdb1"
+export CONFIG_BOOT_DEV="/boot.img"
+export CONFIG_USB_BOOT_DEV="/media.img"
 
 #run: coreboot.intermediate
 run:

build and test run: make BOARD=qemu-coreboot && ./build/make-4.2.1/make BOARD=qemu-coreboot run

[tlaurion]

@osresearch :

It also seems that from Gentoo documentation that it would be possible to pass host random stream by adding: -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0

[tlaurion]

To be tested: https://github.com/tpm2-software/tpm2-tcti-uefi/blob/master/lib/efi-test-setup.sh

[tlaurion]

As pointed in https://github.com/osresearch/heads/issues/700#issuecomment-611628774 by @orangecms : -netdev user,id=u1 -device e1000,netdev=u1 maps the e1000 to host interface and works.

Should be fixed in qemu-coreboot @flammit : have a problem with that fix? Edit: implemented in #707

[tlaurion]

So qemu-coreboot TPM support, which would rely on swtpm support which depends on libtpms support seems possible as dependencies of the qemu-coreboot board, but would be linked to a totally different build path depending if on fedora build system or debian (debian not providing libtpms packages for some reason, requiring to build everything from source)

@orangecms: any success that path?

[tlaurion]

A lot of documentation lies in this ticket for PR propositions. Reopening.

[pietrushnic]

So qemu-coreboot TPM support, which would rely on swtpm support which depends on libtpms support seems possible as dependencies of the qemu-coreboot board, but would be linked to a totally different build path depending if on fedora build system or debian (debian not providing libtpms packages for some reason, requiring to build everything from source)

@orangecms: any success that path?

@tlaurion I'm not sure what qemu-coreboot is, but no matter what emulated TPM using swtpm requires initialization by firmware, which for example is implemented in SeaBIOS. Or maybe Linux kernel TIS/CRB driver is able to bring up TPM from scratch without firmware support - this should be pretty easy to check. Am I missing something?

[tlaurion]

@pietrushnic : qemu-coreboot in Heads term is just the board config

So one can build Heads for qemu calling make BOARD=qemu-coreboot or make BOARD=qemu-coreboot-fbwhiptail and then call make BOARD=qemu-coreboot run or make BOARD=qemu-coreboot-fbwhiptail run and be able to test Heads, with really limited functionnalities, as exposed through #688 and #701 #354 .

Any advice welcome, since building those qemu board configs without them being linked to a software TPM (swtpm), not having functional USB passthrough (to be able to sign /boot components and remotely attest through HOTP) nor assigned prepared block devices/images attached (installed OS or at least a /dev/sda1 to be recognized by Heads as boot partition which is a requirement, sign /boot config with passed USB device) is pretty useless but to show that heads produces a valid ROM only.

qemu-coreboot boards have TPM deactivated, and as reported by issue #688, since no valid block devices are passed to qemu, the board is a diskless machine without proper changes on board configs themselves doing some more magic for the end user calling make BOARD=qemu-coreboot-fbwhiptail run to have measured boot, install an OS and have verified /boot as a Heads user would expect to be able to test out of the box.

@orangecms helped here. Maybe he could share some more light on the subject?

[ReLIFE9527]

@pietrushnic : qemu-coreboot in Heads term is just the board config

So one can build Heads for qemu calling make BOARD=qemu-coreboot or make BOARD=qemu-coreboot-fbwhiptail and then call make BOARD=qemu-coreboot run or make BOARD=qemu-coreboot-fbwhiptail run and be able to test Heads, with really limited functionnalities, as exposed through #688 and #701 #354 .

Any advice welcome, since building those qemu board configs without them being linked to a software TPM (swtpm), not having functional USB passthrough (to be able to sign /boot components and remotely attest through HOTP) nor assigned prepared block devices/images attached (installed OS or at least a /dev/sda1 to be recognized by Heads as boot partition which is a requirement, sign /boot config with passed USB device) is pretty useless but to show that heads produces a valid ROM only.

qemu-coreboot boards have TPM deactivated, and as reported by issue #688, since no valid block devices are passed to qemu, the board is a diskless machine without proper changes on board configs themselves doing some more magic for the end user calling make BOARD=qemu-coreboot-fbwhiptail run to have measured boot, install an OS and have verified /boot as a Heads user would expect to be able to test out of the box.

@orangecms helped here. Maybe he could share some more light on the subject?

So the coreboot.rom or linuxboot.rom built by heads when I calling 'make BOARD=qemu-linuxboot run' is not able to boot the real kernel and root filesystem, since there is no disk can be mounted. Is it possible to build the necessary components separately and then combine them into the linuxboot.rom, and subsequently run it on QEMU with the real kernel and root filesystem? Because I found QEMU can directly run the real kernel and root filesystem without linuxboot.rom, but when I added it, it would be stuck in the recovery shell, just as same as ‘make BOARD=qemu-linuxboot run’. Or, do you have some other good way to solve this problem? Sincerely thanks a lot.

[tlaurion]

@ReLIFE9527 : I was successful applying some hacks here while some help would be required to pass disks (or img) from the host to qemu so that the original mappings would be valid without hacking Heads board configurations but by having a make run statement that works out of the box successfully: mapping usb devices (HOTP tests) with a swtpm being setuped correctly in a precedent step, with its required dependencies.

I am on unknown territories here. @pietrushnic any advices or name tagging here that would be helpful, or any link more specific then the previously documented path that someone with more experience could shed some light on?

Edit:

• [ ] a qemu deployment depending on a host swtpm working instance looks like this. To make this work inside of Heads so that a make BOARD=qemu-linuxboot run works out of the box, we need:

  • [ ] swtpm setuped on host calling qemu

  • [ ] usb passthrough (HTOP) needs to be figured out

  • [ ] disk images needs to be mapped from host so that Heads finds a workable /dev/sda1 and can boot from USB/DISK

[tlaurion]

Interesting USB usage tests were done here

[tlaurion]

@ReLIFE9527 @itay-grudev @pietrushnic : important breakthrough in code on the safeboot side for swtpm+qemu communication and board adjustments.

Reimplementing dependencies on qemu boards, Makefile and modules will lead to be able to test heads with TPM (and some additional tinkering would make USB pass-through of USB Security dongle work)

• swtpm support in safeboot
• relative commit

[pietrushnic]

@tlaurion thanks. We discussed that architecture and potential use cases extensively during QubesOS minisummit SRTM talk. Slides are here

[tlaurion]

That was implemented as draft into https://github.com/osresearch/heads/pull/893 (doesn't build correctly as of now)

[tlaurion]

For USB support (HOTP support) https://qemu.readthedocs.io/en/latest/system/devices/usb.html Building instructions for swtpm missing in modules, so that qemu can ask for it and call it directly from make run.