Combining ImageId= and Encrypt=all fails in initrd on mounting /sysroot

[nlacombe42]

Error message

i am getting an error when trying to boot an arch image created by mkosi that has the root partition encrypted with luks:

mkosi project files

you can see the mkosi project files here: https://github.com/nlacombe42/mkosi-bug

here is the content of mkosi.default :

[Distribution]
Distribution=arch
Architecture=x86_64

[Output]
Format=gpt_btrfs
Bootable=true
BootProtocols=uefi
Encrypt=all
Hostname=testhost
ImageId=test-img

[Packages]
Packages=
    base
    linux
    util-linux
    systemd
    bash

[Partitions]
RootSize=3G
ESPSize=256M

mkosi build logs: https://0x0.st/oBzR.log

Host system info

i ran the build inside this arch vm image using qemu and these steps.

$ uname -a
Linux archlinux 5.17.9-arch1-1 #1 SMP PREEMPT Wed, 18 May 2022 17:30:11 +0000 x86_64 GNU/Linux
$ cat /etc/os-release
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
LOGO=archlinux-logo
$ pacman -Qs 'mkosi|systemd'
local/mkosi 12-1
    Build Legacy-Free OS Images
local/systemd 251.1-1
    system and service manager
local/systemd-libs 251.1-1
    systemd client libraries
local/systemd-sysvcompat 251.1-1
    sysvinit compat for systemd

Steps to reproduce

1. execute the following on your host system to get an arch vm image:

#!/usr/bin/env bash

main() {
	imageFilename="Arch-Linux-x86_64-basic-20220526.57805.qcow2"
	imageUrl="https://gitlab.archlinux.org/archlinux/arch-boxes/-/jobs/57805/artifacts/raw/output/$imageFilename"
	verificationFilename="$imageFilename.SHA256"
	verificationFileUrl="$imageUrl.SHA256"
	qemuDiskFilename="arch-qemu-disk.qcow2"

	if [ ! -f "$verificationFilename" ]; then
		curl -o "$verificationFilename" "$verificationFileUrl" || fatal "error downloading verification file"
	fi

	if [ ! -f "$imageFilename" ]; then
		curl -o "$imageFilename" "$imageUrl" || fatal "error downloading image file"
	fi

	rhash -c "$verificationFilename" || fatal "image file does not pass verification"
	cp "$imageFilename" "$qemuDiskFilename" || fatal "error copying image file to qemu disk file"
	qemu-system-x86_64 -machine q35 --accel kvm -m 2G -drive "index=0,media=disk,format=qcow2,file=$qemuDiskFilename"
}

fatal() {
	echo "$*"
	exit 1
}

main "$@"

2. then execute the following in the qemu guest system (username arch password arch) :

sudo su
pacman -Sy git mkosi dosfstools
git clone https://github.com/nlacombe42/mkosi-bug.git
cd ./mkosi-bug/
./build.sh

3. then quit the vm and extract the /home/arch/mkosi-bug/mkosi.output/test-img.raw file from the vm:

qemu-img convert -pO raw arch-qemu-disk.qcow2 arch-qemu-disk.raw
sudo losetup -P -f arch-qemu-disk.raw
sudo mount /dev/loop0p2 /mnt
cp /mnt/home/arch/mkosi-bug/mkosi.output/test-img.raw test-img.raw
sudo umount /mnt
sudo losetup -D

4. then run qemu on the test-img.raw image: qemu-system-x86_64 -machine q35 --accel kvm -m 2G -bios /usr/share/ovmf/x64/OVMF.fd -drive index=0,media=disk,format=raw,file=./test-img.raw

[ghost]

the part that seems like the most important part of journalctl -b afaict (which happends to be the last logs) :

btw when i booted the image, i was not asked for a (luks) password. and from the error logs, my guess is that it tried to mount the luks partition like if it was a normal (non-luks, unencrypted) partition.

if i use the emergency shell to list a few potentially important directory after the fail boot you can see the result below. i am guessing those are the files present on the initrd image. i can't see any fstab or crypttab file.

[ghost]

i have forked mkosi and created an integration test that you can see here that represents this use case.

ps: and i have managed to simplify the test case a lot. looks like the issue happens when specifying an image id as well as luks on all partitions.

[behrmann]

Judging from this discussion in #992 I think I see that dracut doesn't really know what to mount. Since we have an image ID we set root= on the kernel cmdline (due to the logic in #818 pythonized in #923), thus disabling systemd-gpt-auto-generator, but we also don't have an fstab or cryptab to help dracut out.

First off: Let's maybe take a tiny step back and ask two questions:

1. Why are you setting ImageId=? Is the image meant for some sort of golden master deployment?
2. If so, does the image need to be encrypted?

I think if either of these is not needed, the problem will just go away.

That being said, regardless of whether you can let either ImageId= or Encrypt=all go, I don't see a reason why LUKS and (versioned) golden master images should be incompatible (I might be wrong, but I can imagine just having yubikeys or something similar sticking in the machines or it might be TPM-bound or it could be some network-bound decryption story with tang/clevis).

@poettering Do you have any guidance how to marry both concepts?

[ghost]

1. i am building multiple images; and using ImageId to set the output file name and partition name. the images are for personal use as my main os/distro. i want to be able to bundle config and many other personalizations (including code and thich packages to install, etc).
2. i want it to be encrypted for privacy and integrity of my personal data. and in some cases i might put the image on a usb, so security is more important in this case. i know that the esp is not encrypted but it's still better than no encryption at all.

I think if either of these is not needed, the problem will just go away.

i could try without the ImageId, although not ideal since i'd want to be able to set the partition name to distinguish it from other partitions i might create, i might be able to make it work.

That being said, regardless of whether you can let either ImageId= or Encrypt=all go, I don't see a reason why LUKS and (versioned) golden master images should be incompatible [...]

since there was no documentation stating that those 2 configs are mutially exclusive (and that there was no runtime validation against using both of those) i assumed it should have been possible to do this and that it was a bug that it was not working. if there is some technical limitation then it would make sense to add some documentation and runtime validation for this.

[behrmann]

1. i am building multiple images; and using ImageId to set the output file name and partition name. the images are for personal use as my main os/distro. i want to be able to bundle config and many other personalizations (including code and thich packages to install, etc).

If it were just the name of the output file then Output= would be the way to go. If you need the partition labels changed, you could probably to that in a finalise script.

2. i want it to be encrypted for privacy and integrity of my personal data. and in some cases i might put the image on a usb, so security is more important in this case. i know that the esp is not encrypted but it's still better than no encryption at all.

Makes sense, although you can have both without FDE. See Lennart's recent blog post on the topic (dm-verity for integrity, encrypted homes for user data; at least on user machines there's usually little needing privacy outside of a home as user data should only be in there, for servers there's other ways to go about this.)

since there was no documentation stating that those 2 configs are mutially exclusive (and that there was no runtime validation against using both of those) i assumed it should have been possible to do this and that it was a bug that it was not working. if there is some technical limitation then it would make sense to add some documentation and runtime validation for this.

Well, at least I only found out now, because I hadn't considered mixing those two flavours :) I would well like for both to mix, but haven't given it any thought yet how to. As a stop gap a warning would definitely be the way to go, for more I hope for some second opinions on this.

[DaanDeMeyer]

The issue here seems to be that the required tools are not available in the initrd. Anyway, with the migration to systemd-repart, mkosi doesn't handle encryption anymore but delegates this to systemd-repart, so if this is still an issue it should be reported against systemd-repart in the systemd repository, let's close the issue here.