Sign OS-dependent sysexts using an ephemeral key which is baked into the image.

The signed sysexts are generated using systemd-repart and they are built as a DDI with a dm-verity partition and signature partition.

TODO: determine if we enable the CONFIG_DM_VERITY_VERIFY_ROOTHASH_SIG in this PR or separate (@chewi was planning to use this for the kernel move, but the plans have shifted now).

How to use

Just enable any OS dependent sysexts, they should be verified using dm-verity has signature when loading.

Testing done

When you boot the built image, all OS-dependent sysexts (including OEM) should be signed. When booting, the systemd-sysext will merge them and verify the signature automatically. To merge again with explicitly stricter policy (which forces signed sysexts only), you can use the following command: systemd-sysext refresh --image-policy="root=verity+signed+absent:usr=verity+signed+absent"

See related sysext-bakery PR#175

[x] Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
[ ] Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

Aug 01 '25 15:08 danzatt

systemd-repart only gained the ability to apply compression to erofs in v257, but as far as I can tell, this is only configurable through the repart.d files, which aren't used when creating DDIs.

Actually, I got that slightly wrong. The files in /usr/lib/repart.d aren't used when creating a DDI, but the tool internally uses files from /usr/lib/systemd/repart/definitions. I think it might even pick up files from /etc/systemd/repart/definitions. It doesn't seem to be documented, but it looks like you could just add your own configuration here.

Aug 04 '25 15:08 chewi

Test report for 4524.0.0+nightly-20251119-0830 / amd64 arm64

Platforms tested : qemu_uefi-amd64 qemu_update-amd64 qemu_uefi-arm64 qemu_update-arm64

bpf.execsnoop 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

bpf.local-gadget 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.basic 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.cloudinit.basic 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.cloudinit.multipart-mime 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.cloudinit.script 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.disk.raid0.data 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.disk.raid0.root 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.disk.raid1.data 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.disk.raid1.root 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.etcd-member.discovery 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.etcd-member.etcdctlv3 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.etcd-member.v2-backup-restore 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.filesystem 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.flannel.udp 🟢 Succeeded: qemu_uefi-amd64 (1)

cl.flannel.vxlan 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.instantiated.enable-unit 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.kargs 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.luks 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.oem.indirect 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.oem.indirect.new 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.oem.regular 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.oem.regular.new 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.oem.reuse 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.oem.wipe 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.partition_on_boot_disk 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.symlink 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.translation 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v1.btrfsroot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v1.ext4root 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v1.groups 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v1.once 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v1.sethostname 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v1.users 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v1.xfsroot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v2.btrfsroot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v2.ext4root 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v2.users 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v2.xfsroot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v2_1.ext4checkexisting 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v2_1.swap 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.ignition.v2_1.vfat 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.install.cloudinit 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.internet 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.locksmith.cluster 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.network.initramfs.second-boot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.network.iptables 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.network.listeners 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.network.nftables 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.network.wireguard 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.omaha.ping 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.osreset.ignition-rerun 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.overlay.cleanup 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.swap_activation 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.sysext.boot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.sysext.fallbackdownload # SKIP 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.tang.nonroot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.tang.root 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.toolbox.dnf-install 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.tpm.eventlog 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.tpm.nonroot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.tpm.root 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.tpm.root-cryptenroll 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.tpm.root-cryptenroll-pcr-noupdate 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.tpm.root-cryptenroll-pcr-withupdate 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.update.badverity 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.update.payload 🟢 Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

cl.update.payload-boot-part-too-small 🟢 Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

cl.update.reboot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.users.shells 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

cl.verity 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.auth.verify 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.ignition.groups 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.ignition.once 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.ignition.resource.local 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.ignition.resource.remote 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.ignition.resource.s3.versioned 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.ignition.security.tls 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.ignition.sethostname 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.ignition.systemd.enable-service 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.locksmith.reboot 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.locksmith.tls 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.selinux.boolean 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.selinux.enforce 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

coreos.tls.fetch-urls 🟢 Succeeded: qemu_uefi-amd64 (2); qemu_uefi-arm64 (1) ❌ Failed: qemu_uefi-amd64 (1)

Diagnostic output for qemu_uefi-amd64, run 1

    L1: " Error: _cluster.go:125: curl: (35) Recv failure: Connection reset by peer"
    L2: "cluster.go:145: __curl -s -S -m 30 --retry 2 https://start.fedoraproject.org/__ failed: output , status Process exited with status 35_"
    L3: " "
    L4: "  "

coreos.update.badusr 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

devcontainer.docker 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

devcontainer.systemd-nspawn 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

docker.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

docker.btrfs-storage 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

docker.containerd-restart 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

docker.enable-service.sysext 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

docker.lib-coreos-dockerd-compat 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

docker.network-openbsd-nc 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

docker.selinux 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

docker.userns 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

extra-test.[first_dual].cl.update.docker-btrfs-compat 🟢 Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

extra-test.[first_dual].cl.update.payload 🟢 Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

extra-test.[first_dual].cl.update.payload-boot-part-too-small 🟢 Succeeded: qemu_update-amd64 (1); qemu_update-arm64 (1)

kubeadm.v1.32.4.calico.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.32.4.cilium.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.32.4.flannel.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.33.0.calico.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.33.0.cilium.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.33.0.flannel.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.34.1.calico.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.34.1.cilium.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

kubeadm.v1.34.1.flannel.base 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

linux.nfs.v3 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

linux.nfs.v4 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

linux.ntp 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

misc.fips 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

packages 🟢 Succeeded: qemu_uefi-amd64 (2); qemu_uefi-arm64 (2) ❌ Failed: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

Diagnostic output for qemu_uefi-arm64, run 1

    L1: " _packages/sys-block/open-iscsi (30.73s)"
    L2: "cluster.go:125: Unable to find image _ghcr.io/flatcar/targetcli-fb:latest_ locally"
    L3: "cluster.go:125: latest: Pulling from flatcar/targetcli-fb"
    L4: "cluster.go:125: a2318d6c47ec: Pulling fs layer"
    L5: "cluster.go:125: 3d3086a1439f: Pulling fs layer"
    L6: "cluster.go:125: a2318d6c47ec: Verifying Checksum"
    L7: "cluster.go:125: a2318d6c47ec: Download complete"
    L8: "cluster.go:125: 3d3086a1439f: Verifying Checksum"
    L9: "cluster.go:125: 3d3086a1439f: Download complete"
    L10: "cluster.go:125: a2318d6c47ec: Pull complete"
    L11: "cluster.go:125: 3d3086a1439f: Pull complete"
    L12: "cluster.go:125: Digest: sha256:b6cd65db981974e8b74938617218dd023775b969f9a059ced21e6ce6fa4763c1"
    L13: "cluster.go:125: Status: Downloaded newer image for ghcr.io/flatcar/targetcli-fb:latest"
    L14: "cluster.go:125: mke2fs 1.47.3 (8-Jul-2025)"
    L15: "cluster.go:125: Created symlink _/etc/systemd/system/remote-fs.target.wants/iscsi.service_ ??? _/usr/lib/systemd/system/iscsi.service_."
    L16: "cluster.go:145: __sudo /check__ failed: output no /dev/sda device after reboot, status Process exited with status 1_"
    L17: " "
    L18: " _packages/sys-block/open-iscsi (39.54s)"
    L19: "cluster.go:125: Unable to find image _ghcr.io/flatcar/targetcli-fb:latest_ locally"
    L20: "cluster.go:125: latest: Pulling from flatcar/targetcli-fb"
    L21: "cluster.go:125: 92c3b3500be6: Pulling fs layer"
    L22: "cluster.go:125: 48bc879b34ee: Pulling fs layer"
    L23: "cluster.go:125: 92c3b3500be6: Verifying Checksum"
    L24: "cluster.go:125: 92c3b3500be6: Download complete"
    L25: "cluster.go:125: 48bc879b34ee: Verifying Checksum"
    L26: "cluster.go:125: 48bc879b34ee: Download complete"
    L27: "cluster.go:125: 92c3b3500be6: Pull complete"
    L28: "cluster.go:125: 48bc879b34ee: Pull complete"
    L29: "cluster.go:125: Digest: sha256:b6cd65db981974e8b74938617218dd023775b969f9a059ced21e6ce6fa4763c1"
    L30: "cluster.go:125: Status: Downloaded newer image for ghcr.io/flatcar/targetcli-fb:latest"
    L31: "cluster.go:125: mke2fs 1.47.3 (8-Jul-2025)"
    L32: "cluster.go:125: Created symlink _/etc/systemd/system/remote-fs.target.wants/iscsi.service_ ??? _/usr/lib/systemd/system/iscsi.service_."
    L33: "cluster.go:145: __sudo /check__ failed: output no /dev/sda device after reboot, status Process exited with status 1_"
    L34: " "

Diagnostic output for qemu_uefi-amd64, run 1

    L1: " _packages/sys-block/open-iscsi (30.73s)"
    L2: "cluster.go:125: Unable to find image _ghcr.io/flatcar/targetcli-fb:latest_ locally"
    L3: "cluster.go:125: latest: Pulling from flatcar/targetcli-fb"
    L4: "cluster.go:125: a2318d6c47ec: Pulling fs layer"
    L5: "cluster.go:125: 3d3086a1439f: Pulling fs layer"
    L6: "cluster.go:125: a2318d6c47ec: Verifying Checksum"
    L7: "cluster.go:125: a2318d6c47ec: Download complete"
    L8: "cluster.go:125: 3d3086a1439f: Verifying Checksum"
    L9: "cluster.go:125: 3d3086a1439f: Download complete"
    L10: "cluster.go:125: a2318d6c47ec: Pull complete"
    L11: "cluster.go:125: 3d3086a1439f: Pull complete"
    L12: "cluster.go:125: Digest: sha256:b6cd65db981974e8b74938617218dd023775b969f9a059ced21e6ce6fa4763c1"
    L13: "cluster.go:125: Status: Downloaded newer image for ghcr.io/flatcar/targetcli-fb:latest"
    L14: "cluster.go:125: mke2fs 1.47.3 (8-Jul-2025)"
    L15: "cluster.go:125: Created symlink _/etc/systemd/system/remote-fs.target.wants/iscsi.service_ ??? _/usr/lib/systemd/system/iscsi.service_."
    L16: "cluster.go:145: __sudo /check__ failed: output no /dev/sda device after reboot, status Process exited with status 1_"
    L17: " "
    L18: " _packages/sys-block/open-iscsi (39.54s)"
    L19: "cluster.go:125: Unable to find image _ghcr.io/flatcar/targetcli-fb:latest_ locally"
    L20: "cluster.go:125: latest: Pulling from flatcar/targetcli-fb"
    L21: "cluster.go:125: 92c3b3500be6: Pulling fs layer"
    L22: "cluster.go:125: 48bc879b34ee: Pulling fs layer"
    L23: "cluster.go:125: 92c3b3500be6: Verifying Checksum"
    L24: "cluster.go:125: 92c3b3500be6: Download complete"
    L25: "cluster.go:125: 48bc879b34ee: Verifying Checksum"
    L26: "cluster.go:125: 48bc879b34ee: Download complete"
    L27: "cluster.go:125: 92c3b3500be6: Pull complete"
    L28: "cluster.go:125: 48bc879b34ee: Pull complete"
    L29: "cluster.go:125: Digest: sha256:b6cd65db981974e8b74938617218dd023775b969f9a059ced21e6ce6fa4763c1"
    L30: "cluster.go:125: Status: Downloaded newer image for ghcr.io/flatcar/targetcli-fb:latest"
    L31: "cluster.go:125: mke2fs 1.47.3 (8-Jul-2025)"
    L32: "cluster.go:125: Created symlink _/etc/systemd/system/remote-fs.target.wants/iscsi.service_ ??? _/usr/lib/systemd/system/iscsi.service_."
    L33: "cluster.go:145: __sudo /check__ failed: output no /dev/sda device after reboot, status Process exited with status 1_"
    L34: " "

sysext.custom-docker.sysext 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

sysext.custom-oem 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

sysext.disable-containerd 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

sysext.disable-docker 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

sysext.simple 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

systemd.journal.remote 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

systemd.journal.user 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

systemd.sysusers.gshadow 🟢 Succeeded: qemu_uefi-amd64 (1); qemu_uefi-arm64 (1)

Aug 04 '25 15:08 github-actions[bot]

I've just tested using erofs compression and without (using SYSTEMD_REPART_MKFS_OPTIONS_EROFS="-zlz4hc,12 -C65536 -Efragments,ztailpacking"). There seems to be negligible (8MB difference), so I'm not sure if it's worth the double compression.

Uncompressed

flatcar_production_qemu_uefi_image.img 503M (526516224)

# df -h /usr/
Filesystem       Size  Used Avail Use% Mounted on
/dev/mapper/usr 1016M  546M  331M  63% /usr

# btrfs filesystem usage /usr
Overall:
    Device size:		1015.99MiB
    Device allocated:		 684.00MiB
    Device unallocated:		 331.99MiB
    Device missing:		     0.00B
    Device slack:		     0.00B
    Used:			 542.51MiB
    Free (estimated):		 466.62MiB	(min: 466.62MiB)
    Free (statfs, df):		 330.94MiB
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		   2.87MiB	(used: 0.00B)
    Multiple profiles:		        no

Data+Metadata,single: Size:680.00MiB, Used:542.51MiB (79.78%)
   /dev/mapper/usr	 680.00MiB

System,single: Size:4.00MiB, Used:4.00KiB (0.10%)
   /dev/mapper/usr	   4.00MiB

Unallocated:
   /dev/mapper/usr	 331.99MiB

# ls -lah /usr/share/flatcar/sysext/
total 328M
drwxr-xr-x. 1 root root   80 Oct 23 13:44 .
drwxr-xr-x. 1 root root  166 Oct 23 13:47 ..
-rw-r--r--. 1 root root 156M Oct 23 13:44 containerd-flatcar.raw
-rw-r--r--. 1 root root 192M Oct 23 13:44 docker-flatcar.raw

Compressed

flatcar_production_qemu_uefi_image.img 500M (523436032)

# df -h /usr/
Filesystem       Size  Used Avail Use% Mounted on
/dev/mapper/usr 1016M  538M  331M  62% /usr

# btrfs filesystem usage /usr
Overall:
    Device size:		1015.99MiB
    Device allocated:		 684.00MiB
    Device unallocated:		 331.99MiB
    Device missing:		     0.00B
    Device slack:		     0.00B
    Used:			 534.66MiB
    Free (estimated):		 474.64MiB	(min: 474.64MiB)
    Free (statfs, df):		 330.94MiB
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		   2.69MiB	(used: 0.00B)
    Multiple profiles:		        no

Data+Metadata,single: Size:680.00MiB, Used:534.66MiB (78.63%)
   /dev/mapper/usr	 680.00MiB

System,single: Size:4.00MiB, Used:4.00KiB (0.10%)
   /dev/mapper/usr	   4.00MiB

Unallocated:
   /dev/mapper/usr	 331.99MiB

# ls -lah /usr/share/flatcar/sysext/
total 150M
drwxr-xr-x. 1 root root  80 Oct 23 14:12 .
drwxr-xr-x. 1 root root 166 Oct 23 14:15 ..
-rw-r--r--. 1 root root 86M Oct 23 14:12 containerd-flatcar.raw
-rw-r--r--. 1 root root 85M Oct 23 14:12 docker-flatcar.raw

Oct 23 '25 16:10 danzatt

Interesting, thank you! I was just a little surprised by the results, as I thought plain zstd might come out smaller, but I discussed them with Copilot, and it thought they seemed legitimate. I fully agree that the double compression isn't worth it, especially when it only makes 3MB difference to the QCOW2.

Oct 27 '25 15:10 chewi

I think, I've fixed most of the stuff now. To sum it up:

we don't use FS compression for the built-in sysexts (they'll get compressed by btrfs when placed to the /usr filesystem)
switch format of the built-in sysexts from raw squashfs to DDI with erofs
sign all sysexts with ephemeral key (available only during Flatcar build) and place the certificate in /usr/lib/verity.d
- NOTE: this uses userspace verification, this might cause futre problems if we decide to support IPE, so we might have to switch to baking the sysext key into kernel keyring and switch to kernel-space verification
this PR doesn't add much security per se, as the content of /usr, including the built-in sysext has already been verified using dm-verity
- however, if even built-in sysexts are signed, it opens way to enforce signed-only policy system-wide (including bakery/custom sysexts)
- it pulls in three packages erofs-utils, xxhash (dependency of erofs-utils) and virtual/zlib (Gentoo switched to virtual package for zlib)

Nov 10 '25 10:11 danzatt

Can you start GitHub Actions and a Jenkins build (and link it in the PR description)? That would be nice for testing.

Nov 10 '25 11:11 pothos

Just updated the changelog. The GH action finished successfully yesterday and passed all tests, feel free to grab the artifacts and test! I've just rebased the branch onto main, and I am re-running the tests as well as Jenkins job: http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/7008/

Nov 20 '25 17:11 danzatt

Signed OS-dependent sysexts

Sign OS-dependent sysexts using an ephemeral key which is baked into the image.

How to use

Testing done

Test report for 4524.0.0+nightly-20251119-0830 / amd64 arm64

Uncompressed

Compressed