transient `/etc` also mount `/var/etc` as a layer of the composefs/overlayfs

[jcpunk]

I very much like the transient /etc functionality. However, I'd like to persist the ssh host keys generated on the system.

I've got a bare metal workload/target. Replacing the ssh host keys every time the systems reboot leads to a fair bit of frustration. Our security policy requires system secrets (like ssh keys) to be generated and stored only on the system using them. Deploying system secrets from a secret store (Vault, etc) is expressly forbidden for bare metal.

If I could, as root on the image, manually copy the ssh host keys into /var/etc/ and manually worry about getting the permissions correct, having ostree mount /var/etc/ on top of /etc would give me the transient /etc I desire, but save me from frustrating ssh messages. I'd probably whip up a systemd unit to do the work, package it, and put it in my site's image.

I'd expect if I tried to delete a file in /etc that is mounted over from /var/etc that rm would succeed, but the file would return upon reboot as it persists in /var/etc. Similarly, if I tried to replace some file in /etc, it would be replaced in /etc - but not in /var/etc.

In my expected workflow the selinux labels would be attached by ostree but other than that everything (owner/group/mode) would be the responsibility of the admin who puts things in /var/etc.

This could also be helpful for kubelet certificates - ensuring the only node local config elements are the certs used to prove identity.

[cgwalters]

I very much like the transient /etc functionality. However, I'd like to persist the ssh keys generated on the system.

While it's clear from context I'd still explicitly say "ssh host key" as distinct from the traditional usage of "ssh key" = client key.

Anyways yes, we should flesh out our docs and examples around transient-etc more but I think basically anyone who wants to do what you need here can basically indeed create /var/etc (or whatever directory they want) and then have a systemd unit that runs early in boot to copy data from it. Actually that copying can even be done from the initramfs.

Note this issue heavily intersects with https://github.com/containers/bootc/issues/22 although it's trickier because the ssh host keys here are normally machine specific state.

In general I think many use cass for transient-etc can lean more heavily into "network as source of truth" too, with what remains just being bootstrapping.

I think many scenarios like this may have just one secret stored on the system which then can unlock/decrypt further secrets from a server on the network. In this model then you wouldn't have /var/etc/ssh/ssh_host_key, you'd have just that one bootstrap provisioning secret and use it to fetch the host keys for that specific machine.

[jcpunk]

If I built up a simple service to sync files from /var/etc to /etc, is that something you'd like to include as a sort of contrib example or... (ie I'm happy to contribute this if it is useful)

[BoleynSu]

In general I think many use cass for transient-etc can lean more heavily into "network as source of truth" too, with what remains just being bootstrapping.

I think many scenarios like this may have just one secret stored on the system which then can unlock/decrypt further secrets from a server on the network. In this model then you wouldn't have /var/etc/ssh/ssh_host_key, you'd have just that one bootstrap provisioning secret and use it to fetch the host keys for that specific machine.

I am also looking for first party support for such feature. Currently, I implement the below interface for day 2 config management. MachineId basically use ssh_host_key (it can also be TPM) to sign jwt then to request a Config from ConfigProvider. The ConfigProvider can be implemented as fetching configs from remote server or local data store based on MachinedId. The key is that the MachineId should be persisted somewhere. It would be good to have a standard API in ostree/bootc.

type MachineId struct {
	Type     string `json:"type"`
	Content  string `json:"content"`
	Metadata any    `json:"metadata,omitempty"`
}

type File struct {
	Type     string `json:"type"`
	Path     string `json:"path"`
	Content  string `json:"content"`
	Metadata any    `json:"metadata,omitempty"`
}

type Config struct {
	Files []File `json:"files"`
}

type Status struct {
	Status map[string]string `json:"status"`
}

type MachineProvider interface {
	Id() (machinedId MachineId, err error)
}

type ConfigProvider interface {
	Fetch(machinedId MachineId, current Config) (next Config, err error)
}

type StatusProvider interface {
	Push(machineId MachineId, key string, current string) (previous string, err error)
}

[jcpunk]

Thinking about this a bit more. If a sub key was added to [etc] where if etc.transient=true it would add /var/etc to the list of basedirs that would solve things for me. I expect any files that exist in /usr/etc to mask our the files in /var/etc forcing you to really keep only locally unique files there.

Something like etc.transient_include_var_etc would do the trick.

[jcpunk]

FWIW: Fedora 43-rawhide has selinux equivalent setup for /var/etc to match /etc as of July 1.

[jmarrero]

We had a discussion about this on the Fedora-bootc meeting. I am going to take a pass on documenting an example based on @jcpunk needs as detailed on the meeting:

• ssh server keys
• kerberos keytab (/etc/krb5.keytab)
• /etc/machine-id

[jmarrero]

This is my first pass: https://gitlab.com/fedora/bootc/examples/-/merge_requests/80 but I need to make sure it actually works as I expect.

[cgwalters]

Note this issue heavily intersects with https://github.com/bootc-dev/bootc/issues/22 although it's trickier because the ssh host keys here are normally machine specific state.

That said we should definitely support machine specific attached configmaps/extensions.

/etc/machine-id

BTW for this one there's a dedicated systemd.machine_id= kernel command line option. The path of having key bits rooted in the kernel commandline - especially "immutable" data - I think is a good one.

Also in this space for systems booting via UEFI is systemd-stub extensions.

[jcpunk]

Getting the systemd.machine_id= kernel command line option set would be fantastic.