containers
RFE: skip updates if booted container is a superset
In CoreOS we are considering creating different boot media (i.e. bootable disk images) than upgrade media (i.e. the OCI container image in a registry that bootc uses to upgrade the system). The idea here is that we can include some things in the boot media that are only used on first boot that can then be dropped out of the system on upgrade. A good example of this is Ignition, where currently we ship Ignition updates to all upgrading nodes that exist, which is a waste of bandwidth during the transfer and disk space on the upgraded system.
In this scenario we can first create a "core" bootable container build (i.e. the update payload) and then create a "firstboot" bootable container build that is used to generate our boot media. @travier touches on this a bit here.
-> core bootable container (update payload)
|
-----> firstboot bootable container (derives from core, adds firstboot bits)
|
----> bootable disk images are created from the "firstboot" bootable
container but configured to point at the "core" bootable container
registry pullspec for updates
However, we don't want people who grab an image (or AMI, etc) and start up CoreOS (or any Image Mode variant that wants to use a model like this) to immediately have the update mechanism:
- look at the registry and see the booted container image is different than the target container image in the registry
- update the system to it
In order to support something like this we'd need to be a little smarter. An idea here would be supporting a way to mark derived containers as "containing" another image OR as a "superset" of another image.
In this case the update mechanism would look at the container in the registry (the update target) and get the hash (aabbccdd) and then compare that with booted hash (wwxxyyzz), but also the booted "contains" entry (aabbccdd).
If the update target hash is in either the booted hash or "contained" within the booted container the updater will no-op until that is no longer true. Once the update target hash is different the system will update and the firstboot specific bits will no longer be present on the system.
I guess another thing to consider here is how we'd want to represent this in say the bootc status output. I think it would be important enough to represent there the contained container information because essentially the firstboot container is kind of "transient" until the next update happens.
So for example, if someone is reporting a bug, the more important piece of info would be the versioning/hash of the contained container.
Wouldn't CoreOS want to support factory resets rerunning Ignition? Then we'd have to special case re-pulling the provisioning image.
Is having two different images really worth the mental and logical overhead?
There's other ways to implement this of course; an interesting one would he having ignition be something like a sysext or so.
Wouldn't CoreOS want to support factory resets rerunning Ignition? Then we'd have to special case re-pulling the provisioning image.
Indeed. There would be some tradeoff here.
Is having two different images really worth the mental and logical overhead?
Perhaps. It depends on the benefits it brings. It does add complexity for sure.
There's other ways to implement this of course; an interesting one would he having ignition be something like a sysext or so.
Agree. That is an option and it has been discussed, but not discounted. The complexity with that option is we have to modify our tooling that creates boot images (OSBuild under the hood) to know how to add the Ignition sysext to the initramfs (and potentially some pieces to the real root too). Luckily on the non-sysext option front the tooling already knows how to handle container images.
Another thing something this could be useful for would be local layering use cases where you don't to upgrade to the thing in the registry unless it's newer than the thing you've currently deployed is based on.
An idea here would be supporting a way to mark derived containers as "containing" another image OR as a "superset" of another image.
This kind of exists https://github.com/opencontainers/image-spec/blob/06e6b47e2ef69021d9f9bf2cfa5fe43a7e010c81/annotations.md?plain=1#L50 but it's not widely used.
That said of course "strict superset" is pretty simple for us, we can just compare the layers; the image config isn't relevant at runtime.
Anyways though on the overall idea I'm not opposed...but not excited either offhand (though I could be convinced).
Bigger picture especially in Fedora derivatives we already have basically too many ways to install. This would be a lot like a net-new one that overlaps with the others.
And in that vein I think we need to particularly contrast it with the existing "reprovision existing (cloud-init usually) system via bootc install-to-existing root". In that world, the whole initial AMI/qcow2/whatever is the "disposable provisioning environment" and can have whatever tools are desired.
In particular a pattern I'd love to enable better via this is using cloud-init on the first boot to gather data that then gets persisted more "rigorously" to the bootc system (think something like configmaps/secrets)
I dunno if this bootc issue is really the right place to try to discuss such high level things though? It's probably closer to something we should tackle Fedora side and then drive any lower-level technical bits here?
Another thing something this could be useful for would be local layering use cases where you don't to upgrade to the thing in the registry unless it's newer than the thing you've currently deployed is based on.
We discussed this during the Fedora Bootable Containers community meeting yesterday and also out of band afterwards. Assuming https://gitlab.com/fedora/bootc/tracker/-/issues/4 leans on the side of having a bootc "plugin system" (of which dnf would be just one), then potentially this gives room for reusing some of that API so that first boot layer is kind of like the result of a "firstboot plugin" that's immediately turned off on first boot so it doesn't persist.
