Guixsd vs Heads - How to make it work together?

[ghost]

@tlaurion @jfrederickson @daym

Installation complete without any issues. But heads can't boot/sign separate /boot partition. I use simple unencrypted /boot and encrypted / as LUKS. Here is an install script and config.scm. Any ideas why?

(bootloader (bootloader-configuration
                (bootloader grub-bootloader)
                (target "/dev/sda")))
  
(initrd-modules (cons* "i915" %base-initrd-modules))  

(mapped-devices
   (list (mapped-device
          (source (uuid ""))
          (target "box")
          (type luks-device-mapping))))  

(file-systems (append (list (file-system
                               (device "/dev/mapper/box")
                               (mount-point "/")
                               (type "btrfs")
                               (dependencies mapped-devices))
                              (file-system
                               (device "/dev/sda1")
                               (mount-point "/boot")
                               (type "ext4")))
                        %base-file-systems))

rfkill unblock all
cat << EOF > /etc/wpa_supplicant.conf
network={
  ssid=""
  psk=""
}
EOF
herd stop wpa-supplicant
wpa_supplicant -B -i wlp2s0 -c /etc/wpa_supplicant.conf
dhclient -v wlp2s0

modprobe dm_mod
echo -e "o\nn\np\n1\n\n+128M\nn\np\n2\n\n\nt\n2\n8e\nw" | fdisk /dev/sda
cfdisk /dev/sda (/dev/sda1 - flag bootable)
cryptsetup --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-random luksFormat /dev/sda2
cryptsetup luksOpen /dev/sda2 box
mkfs.btrfs -L root /dev/mapper/box
mount LABEL=root /mnt
mkdir -pv /mnt/boot
mkfs.ext4 -L boot /dev/sda1
mount LABEL=boot /mnt/boot/

herd start cow-store /mnt
mkdir /mnt/etc

blkid -s UUID -o value /dev/sda2 >> /mnt/etc/config.scm

nano /mnt/etc/config.scm
guix system init /mnt/etc/config.scm /mnt
reboot

[tlaurion]

@0rb677 can you upload your signature attempt failing since files inside of Heads that are outside of /boot?

We want the best description of the issue without going in every other direction.

[ghost]

Here you are.

Before

After 10th attempt

Finally i hit enter and returns me to default /boot menu or dropped to shell Strange, i never have this issues with Qubes bootloader.

or this

[tlaurion]

@daym any input on how you installed the OS so that required files are found under /boot?

My quick analysis is that guix expects / to already be mounted so that it can find it's /gnu/store/ files in @0rb677 current use case, that may need to be detailed more here?

[jfrederickson]

My quick analysis is that guix expects / to already be mounted so that it can find it's /gnu/store/ files?

That's correct - the kernel and initrd are in /gnu/store (as are all other packages on the system).

Forget exactly how I set everything up but I have a similar setup on my laptop. I've been meaning to see if I can get something working a bit better but for the time being I've been manually running cryptsetup luksOpen/mount/kexec-select-boot... bit annoying but it works at least.

This issue is relevant as well: https://github.com/osresearch/heads/issues/216

[tlaurion]

@daym if you have any working GUIX installation scenario (partition scheme to have /gnu/store availlable to heads /boot?) or proper documentation to refer to, i'm outside of my comfort/understanding zone here (and others too, see referred #216).

Those clarifications would help al lot of other Heads users and willing contributors.

[daym]

Guix System basically has versioned operating systems (i.e. you can go back to all your historical versions). You can select which version to boot in the grub boot menu. That means not just the kernel but everything, all the installed packages, that are used for that boot are selected by that entry.

If we wanted Heads to work with Guix System then we'd probably have to special-case support for Guix.

A shitty verification that does work is to just copy the current operating system's kernel to /boot and have grub refer to it and have Heads read it. But that would not be nice because it would (1) ignore the whole rest of the operating system for Heads' verification and (2) Heads wouldn't let you boot historical operating systems.

@tlaurion: What does Heads aim to do in the case where it fails here? How does the measurement of the kexeced kernel and environment work?

[tlaurion]

@daym: I don't understand how this Guix vestioning system actually works right now.

In a GUIX system, Is the:

1. LUKS partition unlocked, mounted to /, then /boot mounted under /boot?
2. Bootloader aware of the whole OS versioning system? How does it work, how can you select a versioning tuple? Concice explanations with direct links would be greatly appreciated.

Heads, in the current scenario of copying required files under /boot, would just measure the /boot content and inform if there is a changed against signed digest.

[daym]

@tlaurion:

Bootloader aware of the whole OS versioning system?

The bootloader is not really aware of anything special--it's just that every time the guix system is reconfigured, guix overwrites grub's grub.conf to list all the versions ("generations") that now exist. Guix system uses grub's luks support in order to load the stuff from / . That means it will unlock twice (also ask for the passphrase twice), once in grub to load stuff out of / (for example to load Linux, the initrd etc), and once in Linux to mount /.

(The entire thing works just fine with libreboot's Grub, too)

LUKS partition unlocked, mounted to /, then /boot mounted under /boot?

Yes, if there is a separate unencrypted /boot partition specified in the operating-system configuration.

It is only of very limited use since /boot contents won't change--except for grub.cfg. It basically only contains grub extra files like fonts, themes etc.

Bootloader aware of the whole OS versioning system? How does it work, how can you select a versioning tuple?

A version to boot can be selected either by guix system reconfigure or just by selecting an entry in the grub boot menu. But a new version ("generation") is created each time you change the operating-system configuration, for example to install system packages or to install system services.

Heads, in the current scenario of copying required files under /boot, would just measure the /boot content and inform if there is a changed against signed digest.

I see. I don't think that would be useful in Guix system as it is now.

Docs:

https://guix.gnu.org/manual/en/html_node/Using-the-Configuration-System.html

There it says (the only interesting parts):

Instantiating the System

Assuming the operating-system declaration is stored in the my-system-config.scm file, the guix system reconfigure my-system-config.scm command instantiates that configuration, and makes it the default GRUB boot entry (see Invoking guix system).

Speaking of roll-back, each time you run guix system reconfigure, a new generation of the system is created—without modifying or deleting previous generations. Old system generations get an entry in the bootloader boot menu, allowing you to boot them in case something went wrong with the latest generation. Reassuring, no? The guix system list-generations command lists the system generations available on disk. It is also possible to roll back the system via the commands guix system roll-back and guix system switch-generation.

Although the guix system reconfigure command will not modify previous generations, you must take care when the current generation is not the latest (e.g., after invoking guix system roll-back), since the operation might overwrite a later generation (see Invoking guix system).

https://guix.gnu.org/manual/en/html_node/Invoking-guix-system.html

There it says (the only interesting parts):

It also adds a GRUB menu entry for the new OS configuration, and moves entries for older configurations to a submenu—unless --no-grub is passed.

Example for my grub.conf is

# Set 'root' to the partition that contains /gnu/store.
search --file --set /gnu/store/6w0czjlvkx0qpg105bballh3vmmg8j2w-grub-2.04/share/grub/unicode.pf2
...
menuentry "GNU with Linux-Libre 5.4.44" {
  search --file --set /gnu/store/nw16iq8399sigcgf8vbf2qs5nbj4l0bl-linux-libre-5.4.44/bzImage
  linux /gnu/store/nw16iq8399sigcgf8vbf2qs5nbj4l0bl-linux-libre-5.4.44/bzImage --root=/dev/mapper/cryptroot --system=/gnu/store/wll726k5ql3r95mwbl4v8iz4hrhzbwpk-system --load=/gnu/store/wll726k5ql3r95mwbl4v8iz4hrhzbwpk-system/boot irqpoll root_trim=yes iomem=relaxed modprobe.blacklist=pcspkr,snd_pcsp quiet zram.num_devices=4 psmouse.synaptics_intertouch=0
  initrd /gnu/store/44q9w1w1haj5kc3wlpvxpncwzc9pvaqg-raw-initrd/initrd.cpio.gz
}

submenu "GNU system, old configurations..." {
menuentry "GNU with Linux-Libre 5.4.42 (#64, 2020-05-27 19:56)" {
  search --file --set /gnu/store/7f9maqci9lvrbmmq3g7aks2dvk52zhh4-linux-libre-5.4.42/bzImage
  linux /gnu/store/7f9maqci9lvrbmmq3g7aks2dvk52zhh4-linux-libre-5.4.42/bzImage --root=/dev/mapper/cryptroot --system=/var/guix/profiles/system-64-link --load=/var/guix/profiles/system-64-link/boot irqpoll root_trim=yes iomem=relaxed modprobe.blacklist=pcspkr,snd_pcsp quiet zram.num_devices=4 psmouse.synaptics_intertouch=0
  initrd /gnu/store/a0lp6fscyga3vwzafkwsf1yz02b62ha3-raw-initrd/initrd.cpio.gz
}
menuentry "GNU with Linux-Libre 5.4.42 (#63, 2020-05-22 00:18)" {
  search --file --set /gnu/store/7f9maqci9lvrbmmq3g7aks2dvk52zhh4-linux-libre-5.4.42/bzImage
  linux /gnu/store/7f9maqci9lvrbmmq3g7aks2dvk52zhh4-linux-libre-5.4.42/bzImage --root=/dev/mapper/cryptroot --system=/var/guix/profiles/system-63-link --load=/var/guix/profiles/system-63-link/boot irqpoll root_trim=yes iomem=relaxed modprobe.blacklist=pcspkr,snd_pcsp quiet zram.num_devices=4 psmouse.synaptics_intertouch=0
  initrd /gnu/store/a0lp6fscyga3vwzafkwsf1yz02b62ha3-raw-initrd/initrd.cpio.gz
}
menuentry "GNU with Linux-Libre 5.4.40 (#62, 2020-05-14 19:12)" {
  search --file --set /gnu/store/am92pydjrby5wv2hpvsqara203jrna2c-linux-libre-5.4.40/bzImage
  linux /gnu/store/am92pydjrby5wv2hpvsqara203jrna2c-linux-libre-5.4.40/bzImage --root=/dev/mapper/cryptroot --system=/var/guix/profiles/system-62-link --load=/var/guix/profiles/system-62-link/boot irqpoll root_trim=yes iomem=relaxed modprobe.blacklist=pcspkr,snd_pcsp quiet zram.num_devices=4 psmouse.synaptics_intertouch=0
  initrd /gnu/store/gmp2ms3cy45jn7p6sl0bk6wn1c4dighq-raw-initrd/initrd.cpio.gz
}
menuentry "GNU with Linux-Libre 5.4.40 (#61, 2020-05-12 18:52)" {
  search --file --set /gnu/store/am92pydjrby5wv2hpvsqara203jrna2c-linux-libre-5.4.40/bzImage
  linux /gnu/store/am92pydjrby5wv2hpvsqara203jrna2c-linux-libre-5.4.40/bzImage --root=/dev/mapper/cryptroot --system=/var/guix/profiles/system-61-link --load=/var/guix/profiles/system-61-link/boot irqpoll root_trim=yes iomem=relaxed modprobe.blacklist=pcspkr,snd_pcsp quiet zram.num_devices=4 psmouse.synaptics_intertouch=0
  initrd /gnu/store/fwhx7k1bp1rvbqy3ifm9mwjgisiisn6y-raw-initrd/initrd.cpio.gz
}
...

The hashes are hashes of all the build inputs of that respective thing (that is: source code contents).

Here you can see that the kernel used is not always different, but the system used is (see "--system=").

If using Heads, one wouldn't really have to use Grub to begin with, right? We could just make Heads do the cryptsetup.

[tlaurion]

Far from having a complete understanding of it, but doing small steps over QubesOS debian-10 based TemplateBasedAppVM

[tlaurion]

@daym: want to join the slack channel?

[tlaurion]

OK @daym.

Basically for Heads to work with Guix, guix needs to be installed with a separate /boot partition, and ideally the root filesystem being in a luks container.

Then if my understanding is right, this is the same issue as for Nix? #1001?

[tlaurion]

@daym

Sorry, read in diagonal. Now I understand. That would not work as of now for Heads.

Quick explanation:

• Heads creates a digest, and detach sign everything under /boot. That is, it expects to have files under /boot that don't change often. (/boot/kexec_hashes.txt)
• Even if disk decryption was engaged when discovering that the whole disk was encrypted and mounting it under boot, the same above would apply. Heads would create a digest of everything /boot. Since that filesystem will change, that would not work.
• searching / for kernel initrd and grub.cfg would also search for the whole filesystem.

So here, we would need to think outside of the box.... But I do not see a good solution.

• When no boot partition is found, Heads currently prompts to install or define a boot partition. We could hook on that to build on that and permit to select an encrypted partition. Fine.
  • We could then decrypt and mount that to /boot. No problem there, still.
• Then detach signing, rollback counters for HOTP and kexec_* files would be dropped to what Guix considers /. Ugly but not so bad.
• Then comes the part to create digest and detach sign it.... But digest of what?
  • Where is grub.cfg? under /boot/grub2/grub.cfg still? We could definitely and should measure that.
  • /gnu/store could be all part of the digest and detached signed. You see, that can take a while.... Otherwise, what should be measured?

@tlaurion: What does Heads aim to do in the case where it fails here? How does the measurement of the kexeced kernel and environment work?

If using Heads, one wouldn't really have to use Grub to begin with, right? We could just make Heads do the cryptsetup.

• Heads parses (kexec-parse-boot script) grub.cfg, where my confusion, mixing expected normal /boot (sorry) and NixOS similarities.
• It generates boot entries (kexec-select-boot) for the user to select, just like the actual grub does (in a /boot/kexec_default.*.txt file)
• It then prompts the user to define a boot default kexec-save-default) optionally asking to add a TPM Disk encryption key (kexec-save-key) which is randomly generated, and sealed in TPM NV memory with measurements of firmware, measurements of kernel drivers currently loaded, user fused configs (public key, config override (kexec-seal-key).
  • It then generates a detached signature of all the kexec_* files under kexec.sig, which is verified on default boot against public key fused in the rom.

As you see, the optional TPM Disk encryption key would not make much sense, since the drive would already be unlocked.... And yet again, since Heads is based on kexec, and cannot really pass the encryption passphrase (it could but yet again ugly), booting into Guix would require yet again the user to type a the disk encryption key passphrase.

So yeah. Needs a bit of thinking still. Will revisit #216 as well. Revisited here because of quick fix found for #1001. Do you have a x230? Are you under Matrix?

Yet again, I do not really understand where grub is installed. I will have to install and check for myself... But yeah. We are not there yet. But I would love to see Guix work on Heads. And Heads built on Guix....

[tlaurion]

https://github.com/osresearch/heads/issues/216#issuecomment-647827800

Answered the question. So

• /boot has grub stuff, basically.
• So heads would drop its stuff under /boot, but would need to still open luks container to find kernel, initrd etc.... And then kexec into an excrypted container. The way this can work is if initrd and kernel is launched specifying the UUIDs of the filesystem, so that the OS booted can find itself again... My experience on kexec'ing from encrypted storage was never sucessful as of now.

Grub does it, but without kexec'ing from one kernel to another one. I'm not sure how we can pass the secret here. Any insight?