QEMU testing with OS, TPM, USB token

[JonathonHall-Purism]

I was able to get Heads working pretty well in qemu using a software TPM and USB-forwarded token. Booting, HOTP, TOTP, TPM reset, etc. all work. I think this is a pretty useful way to iterate on Heads, once it is set up it is pretty easy to rebuild and retest.

A remaining limitation is that Heads can't reflash itself from qemu, which means it can't update the GPG keyring. There's now a way to inject a GPG public key at build time to work around this. This is probably solvable, qemu does seem to support writing to flash, I haven't looked into the details of supporting this in flashrom though.

There is no NVRAM currently either, qemu only supports 8 MB of flash space, and the Heads image uses all of it. It is probably possible to extend this in qemu, but I haven't looked into it yet.

Bootstrapping the entire setup from scratch is possible, though there are a number of steps, which I documented in README.md. Some steps could be varied if you already have an OS installed via SeaBIOS, or already have a GPG key to import, but I didn't write about all the variations to try and keep one clear path in the doc.

[tlaurion]

First things first, welcome @JonathonHall-Purism !

There is no NVRAM currently either, qemu only supports 8 MB of flash space, and the Heads image uses all of it. It is probably possible to extend this in qemu, but I haven't looked into it yet.

It is possible to select another image size ffrom coreboot config files. One of the old reason why it was locked into 8mb was to mimic lower sized boards chips to detect build breakage when including new stuff/migrating to newer Linux versions etc. I do not think this is a problem anymore and could be increased.

There's now a way to inject a GPG public key at build time to work around this. This is probably solvable, qemu does seem to support writing to flash, I haven't looked into the details of supporting this in flashrom though.

Awesome that public key can be included. I tried to find references for qemu internal reflashing but didn't find any hints to accomplish this.

Bootstrapping the entire setup from scratch is possible, though there are a number of steps, which I documented in README.md. Some steps could be varied if you already have an OS installed via SeaBIOS, or already have a GPG key to import, but I didn't write about all the variations to try and keep one clear path in the doc.

Will try to replicate the results in the next coming days and comment back!

[JonathonHall-Purism]

Thanks @tlaurion ! I'm excited to contribute to the project :100:

(re: ROM size) I do not think this is a problem anymore and could be increased.

The problem there isn't the rom size in coreboot - it's that qemu is not able to map more than 8 MB of flash space. We'd have to raise that limit in qemu (which I would guess is probably not too difficult, but devil is in the details I'm sure). Currently if you try to enable NVRAM you get an error like "combined size of system firmware exceeds 8388608 bytes".

Or we could decrease the coreboot ROM size to make room for NVRAM, but there is not much free space right now (I'm showing ~170 KB free). I figure we will probably run into the 8 MB ceiling in qemu sooner or later anyway, but there's definitely more I need to understand about this to have the complete picture.

Will try to replicate the results in the next coming days and comment back!

Amazing, let me know how it goes!

[JonathonHall-Purism]

Improved swtpm/libtpms build instructions for Debian - Tray provided instructions to build deb packages, which plays much more nicely with AppArmor (thanks Tray!)

[JonathonHall-Purism]

Simplified the "virtual USB flash drive" instructions - Daniel found that I had omitted the step to create the partition table between the dd and kpartx, but since Heads can handle a formatted device with no partition table, that simplifies the setup a lot (no more kpartx at all, thanks Daniel!)

[tlaurion]

Seems like this is moving!

Some questions @JonathonHall-Purism :

• Is KVM required (virtio) or can this skip virsh and virt-manager altogether and be ran as a qemu command on make run as it was previously from board config?

• Also, how hard would but be to create swtpm module so that having that make BOARD run build swtpm and dependencies from source?

Asking because running Qubes here and nesting virt is not supported. So basically postponing until I install another OS on a separated disk.

Let me know, otherwise I will test this asap but delaying because I cannot just... Test this on current setup.

[JonathonHall-Purism]

@tlaurion KVM isn't a hard requirement of libvirt - I think the simplest path would be to continue using libvirt/virt-manager/virsh but let it drive a QEMU VM without KVM when KVM isn't available. It looks like just tweaking the domain type in the XML from 'kvm' to 'qemu' does this. I gave this a quick try inside a VM (so Heads was nested), it's slower but still bearable and does seem to work.

I'd like to see if I can have it pick 'kvm' or 'qemu' automatically since the speed difference is pretty noticeable, but I'd like it to be relatively easy to use on Qubes too.

libvirt buys us quite a bit for this setup IMO, so while I'm sure it's possible to build/drive swtpm and qemu manually, I think we will end up reimplementing a lot of libvirt:

• it knows how to set up the swtpm persistent store and remember where it is
• it knows how to start up swtpm and hook up whatever socket it provides to qemu, and it'll shut it down
• we'd have to give more customization params for the user to tell us how to USB-forward the token into the VM, libvirt has options to do this by device ID or port, etc., and virt-manager is an easy config tool
• similar for a persistent disk, you'd have to create it manually and tell us where it is to attach to the VM

I think to make run a relatively-persistent config with the same TPM/disk/token could end up requiring a small boatload of variables :cry:

[tlaurion]

I confirm TPM functional, was able to replay instructions to create swtpm and libtpms packages and install, with some comments for review.

Also, from Qubes, I had to reduce memory and change network settings to try to approximate what was under make run, with no perfect result to test deployed xml:

diff --git a/boards/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail.xml b/boards/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail.xml
index ce8fa755..15ce22d0 100644
--- a/boards/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail.xml
+++ b/boards/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail.xml
@@ -1,7 +1,7 @@
-<domain type='kvm'>
+<domain type='qemu'>
   <name>heads-qemu-coreboot-fbwhiptail</name>
-  <memory unit='KiB'>6291456</memory>
-  <currentMemory unit='KiB'>6291456</currentMemory>
+  <memory unit='KiB'>1000000</memory>
+  <currentMemory unit='KiB'>1000000</currentMemory>
   <vcpu placement='static'>2</vcpu>
   <os>
     <type arch='x86_64' machine='q35'>hvm</type>
@@ -14,9 +14,10 @@
   </features>
   <devices>
     <emulator>/usr/bin/qemu-system-x86_64</emulator>
-    <interface type='network'>
-      <source network='default'/>
-      <model type='virtio'/>
+    <interface type="user">
+      <mac address="52:54:00:6f:ab:33"/>
+      <model type="e1000"/>
+      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
     </interface>
     <tpm model='tpm-tis'>
       <backend type='emulator' version='1.2'/>

~It is to note that Qubes doesn't pass complete USB controllers, but usb devices to VMs. I was not able to pass Nitrokey Pro to have it seen by qemu. Were your tests with qemu giving any different results @JonathonHall-Purism ?~ I was able to add controller, my bad

It is to note that https://github.com/osresearch/heads/commit/2715851916e953f3a57c41d6d053d92b0fe3bf3c#diff-d68adaf3548aa4d283e0063fc497bc0bee572e9b64cafec9a15a8bcc428b351cR64-R86 from https://github.com/osresearch/heads/pull/893 was approaching a correct make run from qemu, but swtpm and libtpm were not made as modules which made the process complicated (which you resolved per documentation).

Some other insights from Trammel in this commit for good qemu passed arguments: https://github.com/osresearch/heads/commit/2438db5f11b0fc4395f35f9ce6190fae050305e0, if that is of any help.

Any insights?

[tlaurion]

@JonathonHall-Purism

Some input for qemu/kvm network config

It seems that for e1000e to be functional, we need to specify a user->virtualization mapping otherwise Intel e1000 controller is not found (you can try this through network-init-recovery script from Heads recovery shell)

The e1000e driver is loaded but nothing is mapped under qemu, since qemu is not finding the controller. To make this work:

diff --git a/boards/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail.xml b/boards/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail.xml
index ce8fa755..8d53d461 100644
--- a/boards/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail.xml
+++ b/boards/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail.xml

@@ -1,7 +1,7 @@
-<domain type='kvm'>
+<domain type='qemu'>
   <name>heads-qemu-coreboot-fbwhiptail</name>
-  <memory unit='KiB'>6291456</memory>
-  <currentMemory unit='KiB'>6291456</currentMemory>
+  <memory unit='KiB'>1000000</memory>
+  <currentMemory unit='KiB'>1000000</currentMemory>
   <vcpu placement='static'>2</vcpu>
   <os>
     <type arch='x86_64' machine='q35'>hvm</type>
@@ -14,9 +14,9 @@
   </features>
   <devices>
     <emulator>/usr/bin/qemu-system-x86_64</emulator>
-    <interface type='network'>
-      <source network='default'/>
-      <model type='virtio'/>
+    <interface type="user">
+      <model type="e1000"/>
+      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
     </interface>
     <tpm model='tpm-tis'>
       <backend type='emulator' version='1.2'/>

Also, the current coreboot configuration is not activating measured boot at all. Consequently, cbmem -L is not providing TCPA log, and you can also see that PRC2 is not populated either (cat /sys/class/tpm/tpm0/pcrs).

To make measured boot work

diff --git a/config/coreboot-qemu-fbwhiptail.config b/config/coreboot-qemu-fbwhiptail.config
index 5952dad0..0aed2d6d 100644
--- a/config/coreboot-qemu-fbwhiptail.config
+++ b/config/coreboot-qemu-fbwhiptail.config
@@ -10,6 +10,8 @@ CONFIG_PCIEXP_ASPM=y
 CONFIG_PCIEXP_COMMON_CLOCK=y
 CONFIG_UART_PCI_ADDR=0
 CONFIG_DRIVERS_PS2_KEYBOARD=y
+CONFIG_USER_TPM1=y
+CONFIG_TPM_MEASURED_BOOT=y
 CONFIG_DEFAULT_CONSOLE_LOGLEVEL_6=y
 CONFIG_PAYLOAD_LINUX=y
 CONFIG_PAYLOAD_FILE="../../build/qemu-coreboot-fbwhiptail/bzImage"

As you can see, coreboot configuration needs to know which TPM standard to use. (TPM2 would require in the future: CONFIG_USER_TPM2=y and also a change in xml file for virt-manager to deal correctly with TPM implementation as well)

Maybe we should leave current qemu and qemu-fbwhiptail configurations alone and create -tpm1 new coreboot and board configurations? And later on have -tpm2 board additions?

Also, qemu-coreboot-fbwhiptail (which had no tpm and no HOTP support) now requires both. Consequently, for the sake of this actual PR, I think it would be a good idea to add qemu-coreboot-fbwhiptail-tpm1-hotp and qemu-coreboot-tpm1-hotp, or only qemu-coreboot-fbwhiptail-tpm1-hotp and its associated coreboot configuration file?

Otherwise, non-fbwhiptail board could have -tpm1 but not hotp, but requiring both closes in to only test Nitrokey/Purism Librem Key (HOTP support) which would not cover TOTP base use case.

Thoughts?

[tlaurion]

Also, on qemu (Qubes OS qube) to add USB dongle, user has to:

• pass usb device to qube
• open virt-manager's vm
• click "Show virtual hardware details"
• right click, select add new hardware, go to "USB Host device" and select Nitrokey/Purism device

Then USB Security dongle is accessible.

[tlaurion]

Also note that using virt-manager implies expected peristence of data under /var/lib/libvirt. This is not the case under Qubes, where only /home /usr/local and bind-dirs have persistence.

[tlaurion]

@JonathonHall-Purism :

Here is the board config and coreboot config I used to successfully use swtpm based on prior @osresearch referred work.

As current, I cannot successfully pass USB thumb drive to qemu, while the passing Nitrokey from host to qube through usb-proxy exposes the device correctly.

Note here addition of qubes_run statement in board config:

diff --git a/boards/qemu-coreboot-fbwhiptail-tpm1-hotp/qemu-coreboot-fbwhiptail-tpm1-hotp.config b/boards/qemu-coreboot-fbwhiptail-tpm1-hotp/qemu-coreboot-fbwhiptail-tpm1-hotp.config
new file mode 100644
index 00000000..9502b25b
--- /dev/null
+++ b/boards/qemu-coreboot-fbwhiptail-tpm1-hotp/qemu-coreboot-fbwhiptail-tpm1-hotp.config
@@ -0,0 +1,113 @@
+# Configuration for building a coreboot ROM that works in
+# the qemu emulator in GUI mode thanks to FBWhiptail
+#
+# TPM can be used with a qemu software TPM (TIS, 1.2).  A Librem Key or
+# Nitrokey Pro can also be used by forwarding the USB device from the host to
+# the VM.
+export CONFIG_COREBOOT=y
+export CONFIG_COREBOOT_VERSION=4.13
+export CONFIG_LINUX_VERSION=4.14.62
+
+CONFIG_COREBOOT_CONFIG=config/coreboot-qemu-fbwhiptail-tpm1-hotp.config
+CONFIG_LINUX_CONFIG=config/linux-qemu.config
+
+ifeq "$(CONFIG_UROOT)" "y"
+CONFIG_BUSYBOX=n
+else
+CONFIG_KEXEC=y
+CONFIG_QRENCODE=y
+CONFIG_TPMTOTP=y
+CONFIG_POPT=y
+CONFIG_FLASHTOOLS=y
+CONFIG_FLASHROM=y
+CONFIG_PCIUTILS=y
+CONFIG_UTIL_LINUX=y
+CONFIG_CRYPTSETUP2=y
+CONFIG_GPG2=y
+CONFIG_LVM2=y
+CONFIG_MBEDTLS=y
+CONFIG_DROPBEAR=y
+CONFIG_MSRTOOLS=y
+CONFIG_HOTPKEY=y
+
+#Uncomment only one of the following block
+#Required for graphical gui-init (FBWhiptail)
+CONFIG_CAIRO=y
+CONFIG_FBWHIPTAIL=y
+#
+#text-based init (generic-init and gui-init)
+#CONFIG_NEWT=y
+#CONFIG_SLANG=y
+
+endif
+
+export CONFIG_LINUX_USB_COMPANION_CONTROLLER=y
+CONFIG_LINUX_USB=y
+CONFIG_LINUX_E1000=y
+
+#Uncomment only one BOOTSCRIPT:
+#Whiptail-based init (text-based or FBWhiptail)
+export CONFIG_BOOTSCRIPT=/bin/gui-init
+#
+#text-based original init:
+#export CONFIG_BOOTSCRIPT=/bin/generic-init
+
+export CONFIG_TPM=y
+
+export CONFIG_BOOT_DEV="/dev/sda1"
+export CONFIG_BOARD_NAME="qemu-coreboot-fbwhiptail-tpm1-hotp"
+
+#borrowed from https://github.com/orangecms/webboot/blob/boot-via-qemu/run-webboot.sh
+TPMDIR=$(build)/$(BOARD)/vtpm
+qubes_run:
+	@mkdir -p "$(TPMDIR)"
+	swtpm socket \
+		--tpmstate dir="$(TPMDIR)" \
+		--flags "startup-clear" \
+		--ctrl type=unixio,path="$(TPMDIR)/sock" &
+	sleep 0.5
+
+	-qemu-system-x86_64 \
+		--machine q35 \
+		-m 1G \
+		--serial /dev/tty \
+		--bios $(build)/$(BOARD)/$(CB_OUTPUT_FILE) \
+		-object rng-random,filename=/dev/urandom,id=rng0 \
+		-device virtio-rng-pci,rng=rng0 \
+		-netdev user,id=u1 -device e1000,netdev=u1 \
+		-chardev socket,id=chrtpm,path="$(TPMDIR)/sock" \
+		-tpmdev emulator,id=tpm0,chardev=chrtpm \
+		-device tpm-tis,tpmdev=tpm0 \
+		-usb -device usb-host,hostbus=1,hostport=1 \
+
+	stty sane
+
+# Provision a persistent VM using this firmware and a software TPM - create the
+# VM, or if it already exists, update the ROM configuration.  The machine can
+# then have a disk attached in order to boot an OS, and a USB token can be
+# forward into the guest for HOTP support.
+#
+# This VM is configured with libvirt rather than invoked ad-hoc, so that
+# libvirt will manage the swtpm invocation and persistent disks can be
+# configured easily.
+
+# Default for virsh is qemu:///session, but default for virt-manager is
+# qemu:///system
+LIBVIRT_CONNECTION ?= qemu:///system
+
+# Use the GPG-injected ROM if a key was given, since we can't reflash a GPG
+# keyring in QEMU.  Otherwise use the plain ROM, some things can still be tested
+# that way without a GPG key.
+ifneq "$(PUBKEY_ASC)" ""
+QEMU_BOOT_ROM := $(build)/$(BOARD)/$(CB_OUTPUT_FILE_GPG_INJ)
+else
+QEMU_BOOT_ROM := $(build)/$(BOARD)/$(CB_OUTPUT_FILE)
+endif
+
+provision: $(QEMU_BOOT_ROM)
+	if ! virsh -c "$(LIBVIRT_CONNECTION)" domuuid heads-qemu-coreboot-fbwhiptail 2>/dev/null; then \
+		virsh -c "$(LIBVIRT_CONNECTION)" define "boards/$(BOARD)/heads-qemu-coreboot-fbwhiptail.xml"; \
+	fi
+	virsh -c "$(LIBVIRT_CONNECTION)" dumpxml heads-qemu-coreboot-fbwhiptail \
+		| xmlstarlet ed --update /domain/os/loader --value "$(QEMU_BOOT_ROM)" \
+		| virsh -c "$(LIBVIRT_CONNECTION)" define /dev/stdin
diff --git a/config/coreboot-qemu-fbwhiptail-tpm1-hotp.config b/config/coreboot-qemu-fbwhiptail-tpm1-hotp.config
new file mode 100644
index 00000000..6bf546f6
--- /dev/null
+++ b/config/coreboot-qemu-fbwhiptail-tpm1-hotp.config
@@ -0,0 +1,18 @@
+# CONFIG_INCLUDE_CONFIG_FILE is not set
+CONFIG_CBFS_SIZE=0x780000
+# CONFIG_POST_IO is not set
+# CONFIG_POST_DEVICE is not set
+CONFIG_BOARD_EMULATION_QEMU_X86_Q35=y
+# CONFIG_CONSOLE_SERIAL is not set
+CONFIG_LINUX_COMMAND_LINE="debug"
+CONFIG_COREBOOT_ROMSIZE_KB_8192=y
+CONFIG_PCIEXP_ASPM=y
+CONFIG_PCIEXP_COMMON_CLOCK=y
+CONFIG_UART_PCI_ADDR=0
+CONFIG_DRIVERS_PS2_KEYBOARD=y
+CONFIG_USER_TPM1=y
+CONFIG_TPM_MEASURED_BOOT=y
+CONFIG_DEFAULT_CONSOLE_LOGLEVEL_6=y
+CONFIG_PAYLOAD_LINUX=y
+CONFIG_PAYLOAD_FILE="../../build/qemu-coreboot-fbwhiptail-tpm1-hotp/bzImage"
+CONFIG_LINUX_INITRD="../../build/qemu-coreboot-fbwhiptail-tpm1-hotp/initrd.cpio.xz" 

Waiting for your comments on:

• [ ] code review
• [ ] insights on feasibility to pass kvm to qemu (maybe have different configs and board statements would do) and
• [ ] fixes/tests of proposed xml network config as opposed to using (non-always) existing default.
• [ ] /var non-persistence over Qubes OS unless the qcow disk image is created also under board dir, just like the swtpm state dir is created in board config proposed above (we can take for granted that that directory is under persistent storage).
• [ ] Any hint on how to pass usb storage device to qemu directly (see below)? Otherwise, I will work on that later on. The final passed qemu (which can boot from USB device) looks like this, and this is the farther I was able to get today in my testings: /usr/bin/qemu-system-x86_64 -name guest=heads-qemu-coreboot-fbwhiptail,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-heads-qemu-coreboot-/master-key.aes -blockdev {"driver":"file","filename":"/home/user/heads/build/qemu-coreboot-fbwhiptail/heads-qemu-coreboot-fbwhiptail-v0.2.1.bis2-105-g04a46efc-dirty.rom","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"} -machine pc-q35-5.2,accel=tcg,usb=off,smm=on,dump-guest-core=off,pflash0=libvirt-pflash0-format,memory-backend=pc.ram -cpu qemu64 -m 977 -object memory-backend-ram,id=pc.ram,size=1024458752 -overcommit mem-lock=off -smp 2,sockets=2,cores=1,threads=1 -uuid b6731524-d821-4050-8273-2af7913df36d -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=34,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 -device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 -device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2 -device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3 -device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4 -device qemu-xhci,id=usb,bus=pci.1,addr=0x0 -netdev user,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:da:1f:19,bus=pcie.0,addr=0x3 -tpmdev emulator,id=tpm-tpm0,chardev=chrtpm -chardev socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/1-heads-qemu-coreboot--swtpm.sock -device tpm-tis,tpmdev=tpm-tpm0,id=tpm0 -spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on -device bochs-display,id=video0,vgamem=16384k,bus=pcie.0,addr=0x1 -device usb-host,hostdevice=/dev/bus/usb/001/009,id=hostdev0,bus=usb.0,port=1 -device usb-host,hostdevice=/dev/bus/usb/002/011,id=hostdev1,bus=usb.0,port=3 -device virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x0 -object rng-random,id=objrng0,filename=/dev/urandom -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.5,addr=0x0 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on

Which seems to map -device usb-host with the host-device directly (still qubes here, so can be generalizable). Snippet from above: -device usb-host,hostdevice=/dev/bus/usb/001/009,id=hostdev0,bus=usb.0,port=1 -device usb-host,hostdevice=/dev/bus/usb/002/011,id=hostdev1,bus=usb.0,port=3

[JonathonHall-Purism]

Amazing, thanks for all the testing and input @tlaurion ! Lots of items to address, I'm going to run through this on Qubes myself to get all the feedback integrated.

1. Code review - Will fix those errors, thanks! :facepalm:
2. TPM/key in qemu - Yeah I agree let's make this a separate board config, I will split it up.
3. /var non-persistence - OK, I see where you're going with the bare-qemu approach, using fixed locations for persistence in the board directory. I did also like the libvirt approach because it offered a lot of flexibility in (re-)configuring the VM, shuffling disks around without reinstalling OSes, etc., but maybe I can do that with some scripts and symlinks instead. I'll see how far I can get with this approach.
4. USB storage to qemu - I'm not sure offhand how to do this, but I will look at what libvirt does, should offer some insight.

I have a handful of other things needing my attention today, but I'll get back to this soon, just know I haven't forgotten :grin:

[JonathonHall-Purism]

@tlaurion Reworked this quite a bit - eliminated libvirt, and the whole process works in Qubes now. I rewrote the instructions in the README, I think it's a little bit simpler.

PureOS and Debian live LXDE both work and install. GNOME is brutally slow nested under Qubes but it works. However, Debian netinst and Fedora server won't boot with Bochs video (not specific to Heads, although outside of Heads there is a text-based fallback mode menu). Supporting QXL or virtio in Heads would be nice but this is still plenty to test with.

I really wanted to have the 'run' target depend on the actual ROM, but re-running didn't work - the ROM was always altered on re-run so the TOTP secret could never be unsealed. I didn't look into this much yet but I don't think it needs to block this PR.

There's still a sticky detail that if the qemu invocation fails for any reason, swtpm is left running and must be killed manually. Maybe there's some make-fu to take care of that, but it's not that big an issue and the script is fairly complex already.

[JonathonHall-Purism]

I don't think this is related to this PR, looks like the debian:11 image has been updated compared to the last successful builds. Any ideas offhand @tlaurion ?

configure:6738: checking termio.h usability
configure:6738: /root/project/crossgcc/bin/x86_64-linux-musl-gcc -fdebug-prefix-map=/root/project=heads -gno-record-gcc-switches -D__MUSL__ -isystem /root/project/install/include -L/root/project/install/lib  -c -g -O2 -Wall -Wno-pointer-sign -Wpointer-arith  -Wall conftest.c >&5
conftest.c:63:10: fatal error: termio.h: No such file or directory
 #include <termio.h>
          ^~~~~~~~~~

[tlaurion]

The cache couldn't be restored https://app.circleci.com/pipelines/github/osresearch/heads/431/workflows/0c946ab1-673e-4512-a877-9f3ed9e417e2/jobs/4076/parallel-runs/0/steps/0-108

@JonathonHall-Purism Go under circleci project settings, environment variable

And add a new CACHE_VERSION variable so that you start a fresh build without cache, as per CircleCI cache settings:

https://github.com/osresearch/heads/blob/master/.circleci/config.yml#L80

[tlaurion]

It also failed trying to download https://app.circleci.com/pipelines/github/osresearch/heads/431/workflows/0c946ab1-673e-4512-a877-9f3ed9e417e2/jobs/4077/parallel-runs/0/steps/0-103

The error is on the board build, where all logs are outputted in the logs at next step.

I would simply add environment variable to retry1 as indicated on previous step and retry the build.

[tlaurion]

I just pushed the retry button to see if error is reproducible

[tlaurion]

Hmm first time that I have access denied on cache...

https://app.circleci.com/pipelines/github/osresearch/heads/431/workflows/c97b3e67-f9a3-4f8e-b365-09be60e063c9/jobs/4078/parallel-runs/0/steps/0-108

Will change variable and trigger rebuild

[JonathonHall-Purism]

matt.ucc.asn.au seems to be completely down for dropbear, but there is a mirror: https://mirror.dropbear.nl/mirror/releases/dropbear-2016.74.tar.bz2

[tlaurion]

Prep step running without cache https://app.circleci.com/pipelines/github/osresearch/heads/431/workflows/57899f73-9d46-4578-8244-e9bb1aada9ba/jobs/4080

Hopefully it will build clean.

I think I see what is happening here.

I would advise everyone at purism to follow their clones from circle ci.

This will prevent the builds to consume all free circleci credits of osresearch account and would build on your own personal accounts.

Also, when build errors like this happens, each user can change their environment variables to start from clean builds.

• [ ] Maybe we should document this under heqds-wiki development section

[tlaurion]

matt.ucc.asn.au seems to be completely down for dropbear, but there is a mirror: https://mirror.dropbear.nl/mirror/releases/dropbear-2016.74.tar.bz2

Unfortunately typing from cell phone here away from keyboard. But I'm getting sick of this.

• [x] That would deserve another issue to point all current packages to Debian archive mirrors for all heads packages. Those name resolution issues have been a toll and prevents as well reproducibility of builds for commit over time.

[tlaurion]

@JonathonHall-Purism please rebase on master.

[JonathonHall-Purism]

Thanks, I sent a PR to switch to the mirror for now, I expect this one will still fail until that is merged in. Agree that using the Debian archive would be a good move. I'll check back on these tomorrow.

Thanks for the advice on CircleCI too, I set CACHE_VERSION and configured my fork of heads in CircleCI under my account.

[tlaurion]

@JonathonHall-Purism merged your PR since there is no checksum change implied.

[tlaurion]

@JonathonHall-Purism : Under Qubes debian-11-swtpm template based qube (with built local deb packages qvm-copy'ed there and installed in the template)

make BOARD=qemu-coreboot-fbwhiptail-tpm1-hotp USB_TOKEN=NitrokeyPro run

Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize kvm: No such file or directory

To fix, differentiating kvm and qemu virt options seems the simplest way to go

--- /tmp/aSKKk2_qemu-coreboot-fbwhiptail-tpm1-hotp.config
+++ /home/user/heads/boards/qemu-coreboot-fbwhiptail-tpm1-hotp/qemu-coreboot-fbwhiptail-tpm1-hotp.config
@@ -112,7 +112,7 @@
 QEMU_USB_TOKEN_DEV := -device "usb-host,$(USB_TOKEN)"
 endif
 
-run: $(TPMDIR)/.manufacture $(ROOT_DISK_IMG) $(MEMORY_SIZE_FILE) $(USB_FD_IMG)
+kvm-run: $(TPMDIR)/.manufacture $(ROOT_DISK_IMG) $(MEMORY_SIZE_FILE) $(USB_FD_IMG)
 	swtpm socket \
 		--tpmstate dir="$(TPMDIR)" \
 		--flags "startup-clear" \
@@ -139,3 +139,31 @@
 
 	stty sane
 	@echo
+
+qemu-run: $(TPMDIR)/.manufacture $(ROOT_DISK_IMG) $(MEMORY_SIZE_FILE) $(USB_FD_IMG)
+	swtpm socket \
+		--tpmstate dir="$(TPMDIR)" \
+		--flags "startup-clear" \
+		--terminate \
+		--ctrl type=unixio,path="$(TPMDIR)/sock" &
+	sleep 0.5
+
+	-qemu-system-x86_64 "$(ROOT_DISK_IMG)" \
+		--machine q35 \
+		-m "$$(cat "$(MEMORY_SIZE_FILE)")" \
+		--serial /dev/tty \
+		--bios "$(QEMU_BOOT_ROM)" \
+		-object rng-random,filename=/dev/urandom,id=rng0 \
+		-device virtio-rng-pci,rng=rng0 \
+		-netdev user,id=u1 -device e1000,netdev=u1 \
+		-chardev socket,id=chrtpm,path="$(TPMDIR)/sock" \
+		-tpmdev emulator,id=tpm0,chardev=chrtpm \
+		-device tpm-tis,tpmdev=tpm0 \
+		-device qemu-xhci,id=usb \
+		-device usb-tablet \
+		-drive file="$(QEMU_USB_FD_IMG)",if=none,id=usb-fd-drive,format=raw \
+		-device usb-storage,bus=usb.0,drive=usb-fd-drive \
+		$(QEMU_USB_TOKEN_DEV) \
+
+	stty sane
+	@echo

[JonathonHall-Purism]

@tlaurion accel=kvm:tcg is supposed to automatically fall back to tcg, it works for me in a Debian 11 qube. You should get qemu-system-x86_64: falling back to tcg right after the KVM errors. Is there another error?

I can split them up if you want, but if the fallback works and there was actually another error, it's simpler documentation-wise to have just one run IMO.

[tlaurion]

@JonathonHall-Purism Sorry about false alarm before, my issue was linked to specifying wrong QEMU_USB_TOKEN_DEV (NitrokeyPro vs Nitrokey)

On that note, not sure of what is the valid ID passed for Nitrokey Storage here, but that would be another ID to add in the list.

So no need to seperate qemu/kvm at this point in testing this PR.

[tlaurion]

@JonathonHall-Purism I tried booting qemu with Fedora-36 workstation and now digging down issues with FB, and making my way into validating correct coreboot native init settings + linux config.

At the moment, I can get a cursor on screen at standard booting option (no PXE option) but the machine halts prior of getting anaconda FB.

I've worked around that issue in the past for other board configs when Qubes (Fedora) got upgraded and that needed VBT to be added into coreboot to be able to facilitate DRM helper to get proper graphic init from linux + Xorg.

As of now, I see from here that CONFIG_DRM_CIRRUS_QEMU is deprecated, and that link suggests to have DRM_BOCHS compiled in kernel, which is what we currently have.

Better alternatives are:
- stdvga (DRM_BOCHS, qemu -vga std, default in qemu 2.2+)
- qxl (DRM_QXL, qemu -vga qxl, works best with spice)
- virtio (DRM_VIRTIO_GPU), qemu -vga virtio) 

Testing options and will come back here. For the moment, I am not able to install Fedora-36 to continue tests, boot hanging at gnome systemd startup.

But I wonder what you have tested as ISO inside of qubes with qemu? And if DRM_QXL and DRM_VIRTIO_GPU should be activated under linux configuration as well and what is best for kvm?

[tlaurion]

Test trace:

user@heads-tests:~/heads$ git diff
diff --git a/boards/qemu-coreboot-fbwhiptail-tpm1-hotp/qemu-coreboot-fbwhiptail-tpm1-hotp.config b/boards/qemu-coreboot-fbwhiptail-tpm1-hotp/qemu-coreboot-fbwhiptail-tpm1-hotp.config
index b9dd4adb..93b6ae24 100644
--- a/boards/qemu-coreboot-fbwhiptail-tpm1-hotp/qemu-coreboot-fbwhiptail-tpm1-hotp.config
+++ b/boards/qemu-coreboot-fbwhiptail-tpm1-hotp/qemu-coreboot-fbwhiptail-tpm1-hotp.config
@@ -51,6 +51,10 @@ export CONFIG_BOOTSCRIPT=/bin/gui-init
 #
 #text-based original init:
 #export CONFIG_BOOTSCRIPT=/bin/generic-init
+export CONFIG_BOOT_REQ_HASH=n
+export CONFIG_BOOT_REQ_ROLLBACK=n
+export CONFIG_BOOT_KERNEL_ADD="vga=current"
+export CONFIG_BOOT_KERNEL_REMOVE="quiet"
 
 export CONFIG_TPM=y
 
diff --git a/config/linux-qemu.config b/config/linux-qemu.config
index 207f7a0b..5e62ae41 100644
--- a/config/linux-qemu.config
+++ b/config/linux-qemu.config
@@ -227,6 +227,7 @@ CONFIG_INTEL_PCH_THERMAL=y
 CONFIG_MFD_SYSCON=y
 # CONFIG_VGA_ARB is not set
 CONFIG_DRM=y
+CONFIG_DRM_QXL=y
 CONFIG_DRM_BOCHS=y
 CONFIG_FRAMEBUFFER_CONSOLE=y
 CONFIG_USB_HID=m

dmesg under Heads shows drm and fb0 assigned to bochs kernel drivers being used.

Added board config options to be passed to final kexec call (those config options are exported under /etc/config under ROM and used as default additions/removals passed to final kernel):

• quiet kernel option removal (I think we need to see all output under qemu/kvm)
• vga=current kernel option addition to attempt to reuse fb buffer Recompile (will produce dirty image which is then used by the run statement correctly)

As of now, those are not resolving the issue.

Learning my way into using qemu frontend and compat_monitor0 (where you can pass sendkey ctrl-alt-f3 to get into other terminal, our lovely virt-manager gui was offering those directly)

Quick notes for #516 #701

[tlaurion]

Attempting to boot Fedora with basic graphic support (nomodeset) is not helping on my side.