osbuild: use bootc install to deploy the container

[jbtrystram]

Instead of deploying the container to the tree then copy all the contents to the disk image, use bootc to directly manage the installation to the target filesystems.

Right now this requires to use the image as the buildroot so this requires python (for osbuild). This is tracked in [1].

[1] https://github.com/bootc-dev/bootc/issues/1410 Requires https://github.com/osbuild/osbuild/pull/2149

[openshift-ci[bot]]

Skipping CI for Draft Pull Request. If you want CI signal for your change, please convert it to an actual PR. You can still manually trigger a test run with /test all

[dustymabe]

I switched the CI on this to run against rawhide (contains python) so we could actually test the change.

[dustymabe]

A few diffs picked up by cosa diff --metal from https://github.com/coreos/coreos-assembler/pull/4226

cosa-diff-metal.txt

We should probably profile each diff (maybe in https://github.com/coreos/fedora-coreos-tracker/issues/1827) and evaluate whether it's a change we want to make or not.

[dustymabe]

I can't get a built qemu image to boot. I suspect probably the root= and boot= UUIDs added on the kernel command line?

[jbtrystram]

I can't get a built qemu image to boot. I suspect probably the root= and boot= UUIDs added on the kernel command line?

do you mind sharing more logs ? What I am getting locally is ignition failing on https://github.com/coreos/fedora-coreos-tracker/issues/1250

[dustymabe]

Ahh. I see that too now:

[    4.726843] ignition[875]: Ignition failed: failed to create users/groups: failed to configure users: failed to create user "core": exit status 10: Cmd: "useradd" "--root" "/sysroot" "--create-home" "--password" "*" "--comment" "CoreOS Admin" "--groups" "adm,sudo,systemd-journal,wheel" "core" Stdout: "" Stderr: "useradd: cannot lock /etc/group; try again later.\n"

[jbtrystram]

I just re-tried with a rawhide OCI image out of the pipeline and hitting the same thing.

~I see that systemd-fsck-root.service fails to run but, re-running it manually from the initramfs worked, so maybe it runs too early ?~

neverminds, it was just a weird state that the system is after ignition failed.

[jbtrystram]

Ok interesting I found that in the built with bootc image the var directory is not empty:

$ ~ls qemu-mount/ostree/deploy/default/var                                                                                                                                                      
lib  run  tmp

In the initramfs, links under /sysroot e.g. /sysroot/etc -> var/etc exist but /sysroot/var/etc does not exist. Could that be why adduser fail ? @PeaceRebel is seeing the same issue when doing cosa import a container build I think.

[jbtrystram]

I can't get a built qemu image to boot. I suspect probably the root= and boot= UUIDs added on the kernel command line?

looks like removing those make the boot process go further (ignition completes), and out of the initramfs but fail to mount the boot partition.

[jbtrystram]

Blocked on https://github.com/bootc-dev/bootc/issues/1441

[jbtrystram]

ok this works with the following PRs :

• https://github.com/bootc-dev/bootc/pull/1451
• https://github.com/osbuild/osbuild/pull/2152
• https://github.com/osbuild/osbuild/pull/2149

for the bootc PR, it can be built then added into the image through overrides/rootfs. Make sure to build rawhide.

[jbtrystram]

follow-up : either find a way to get the boot components inside cosa, or change the bootc code to call bootupd from the deployed root . I think the latter is preferable. I filed https://github.com/bootc-dev/bootc/issues/1455

[jbtrystram]

follow-up : either find a way to get the boot components inside cosa, or change the bootc code to call bootupd from the deployed root . I think the latter is preferable. I filed bootc-dev/bootc#1455

Made https://github.com/bootc-dev/bootc/pull/1460 With this, we no longer require to use the container as the buildroot, cosa works, so we could do that on all streams.

[jbtrystram]

Alright, marking this as ready for review as all the bits are in place. I guess i need to update the osbuild manifest or the other arches as well, but I'll do that after a review to reduce the amount of back and forth.

This will need a release of bootc.

[jlebon]

Some comments.

I think there are a few things we need to iron out before we can really move forward with this:

1. supporting both old and new paths at the same time

Do we need to? Usually when we make a change this large we roll it out slowly, which means we have to support both ways for some time.

This PR is ignoring that fact, but TBH looking at OSBuild configs that support both would be pretty intimidating, so I'm not excited about trying to do that either. I'd be interested in @jlebon or @travier's thoughts.

Rolling this out in FCOS first would be preferable indeed. Instead of adding mpp conditionals, could we just have a separate set of manifests to use temporarily? Not ideal either, I know.

If we're sufficiently convinced by the diff, we could do a hard cut over. The problem is in ensuring the diff captures everything well. I think it helps a lot though that the main changes here are happening at the filesystem level.

[jbtrystram]

Rolling this out in FCOS first would be preferable indeed. Instead of adding mpp conditionals, could we just have a separate set of manifests to use temporarily? Not ideal either, I know.

I went up this route : duplicated the manifest and added a switch to the osbuild wrapper to use it when releasever is >44. We can drop this when we are confident it can go to all streams.

I cleaned up the commits a bit and incorporated @dustymabe reviews suggestions. If that approach looks good i'll generalize to other arches than x86 (or maybe not?) in another commit.

[jbtrystram]

Ok I found another quirk : https://github.com/bootc-dev/bootc/issues/1559

A workaround would be to use the container as the osbuild buildroot (but that would only work with rawhide as we need python). This is what I was doing until now so that's why I dind't ran into it before.

But since we are only using this manifest for rawhide for now, why not..

[jbtrystram]

requires https://github.com/coreos/bootupd/pull/990

[jbtrystram]

Rebased and removed osbuild patches as they have landed in an upstream release now (requires a COSA rebuild though)

[dustymabe]

Thank you for your patience here JB. I don't see anything too alarming in the patch set here. Mostly small stuff.

I do want to spend some time with this change testing it out locally and poking around the built images. I'll try to do so tomorrow.

[dustymabe]

A couple of things:

1. Trying to build the live media failed with:

Verifying that repacked image matches digest                                                                                                                                                                                                                                  
Packing successful!                                                                                                                                                                                                                                                           
Kernel binary linked: /run/osbuild/tree/tmp_1cwx4nh/tmp-rootfs-dir/boot/vmlinuz-6.17.0-0.rc7.250924gcec1e6e5d1ab3.58.fc44.x86_64                                                                                                                                              
Kernel HMAC linked: /run/osbuild/tree/tmp_1cwx4nh/tmp-rootfs-dir/boot/.vmlinuz-6.17.0-0.rc7.250924gcec1e6e5d1ab3.58.fc44.x86_64.hmac                                                                                                                                          
Creating erofs with -zlzma,level=6 -Eall-fragments,fragdedupe=inode -C1048576 --quiet                                                  
Traceback (most recent call last):                                                                                                     
  File "/run/osbuild/bin/org.osbuild.coreos.live-artifacts.mono", line 967, in <module>                                                
    r = main(workdir=tmdir,                                                                                                            
             tree=args["tree"],                                                                                                        
             inputs=args["inputs"],                                                                                                    
             options=args["options"],                                                                                                  
             loop_client=remoteloop.LoopClient("/run/osbuild/api/remoteloop"))                                                         
  File "/run/osbuild/bin/org.osbuild.coreos.live-artifacts.mono", line 929, in main                                                    
    blsentry_kargs, osname = mkrootfs_metal(paths, workdir, img_metal, fstype, fsoptions, loop_client)                                 
                             ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                 
  File "/run/osbuild/bin/org.osbuild.coreos.live-artifacts.mono", line 638, in mkrootfs_metal                                          
    with open(os.path.join(tmp_rootfs_dir, ALEPH_FILENAME), encoding='utf8') as f:                                                     
         ~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                           
FileNotFoundError: [Errno 2] No such file or directory: '/run/osbuild/tree/tmp_1cwx4nh/tmp-rootfs-dir/.aleph-version.json'

2. The metal diff did show some differences (attached from-metal-ls-diff.txt and to-metal-ls-diff.txt).

• ./.aleph-version.json and the corresponding ./.coreos-aleph-version.json symlink are now just ./.bootc-aleph.json.
• There are ./boot/efi/EFI/fedora/bootuuid.cfg and ./boot/grub2/bootuuid.cfg that I think in the past we generate on first boot for CoreOS, is that right?
• ./boot/grub2/locale is missing a bunch of files that were there before (not sure if this is important).
• A bunch of new files in ./ostree/bootc/storage/
• The ref for the container seems to now include more information (i.e. about the pullspec) ./ostree/repo/refs/heads/ostree/container/image/docker_3A__2F__2F_quay_2E_io -> ./ostree/repo/refs/heads/ostree/container/image/docker_3A__2F__2F_ostree-image-signed_3A_docker_3A__2F__2F_quay_2E_io

from-metal-ls-diff.txt to-metal-ls-diff.txt

[jbtrystram]

A couple of things: ....

Thank you so much for looking into this Dusty ! another set of eyes is really helpful. I'll try to address these this week

[jbtrystram]

There are ./boot/efi/EFI/fedora/bootuuid.cfg and ./boot/grub2/bootuuid.cfg that I think in the past we generate on first boot for CoreOS, is that right?

See https://github.com/coreos/coreos-assembler/pull/4224#discussion_r2310101959

./.aleph-version.json and the corresponding ./.coreos-aleph-version.json symlink are now just ./.bootc-aleph.json

We'll have to re-add this symlink in a postprocess I guess. I can change stages/org.osbuild.ostree.aleph to have it take an argument that only does the symlink if the aleph already exist. Note that this aleph is generated by bootc and have different content than ours. I guess I need to dig into what we use this for

A bunch of new files in ./ostree/bootc/storage/

Looks like this is a leaked overlay mount, see this todo

[jbtrystram]

Added a commit that fixes the missing aleph file symlink. This also fixes the live artifact build

Requires https://github.com/osbuild/osbuild/pull/2226

[jbtrystram]

@dustymabe : Rebased and updated. I included a manifest knob and a env var to make use of the bootc manifest.

* `./boot/grub2/locale` is missing a bunch of files that were there before (not sure if this is important).

It looks like those come from running grub-mkconfig but our boot partition should be populated with bootupd, so I don't understand how that differ. I could not find any lead in the bootupd code.

* The `ref` for the container seems to now include more information (i.e. about the pullspec) `./ostree/repo/refs/heads/ostree/container/image/docker_3A__2F__2F_quay_2E_io` -> `./ostree/repo/refs/heads/ostree/container/image/docker_3A__2F__2F_ostree-image-signed_3A_docker_3A__2F__2F_quay_2E_io`

Do you know any code that rely on the output of ostree refs ? A quick search didn't yield much

[jbtrystram]

It looks like those come from running grub-mkconfig but our boot partition should be populated with bootupd, so I don't understand how that differ. I could not find any lead in the bootupd code.

OK i dug more on this and it's another instance of https://github.com/bootc-dev/bootc/issues/1559 In the previous manifest we were using a chroot before calling bootupd which made sure bootupd (and by extension grub2) would pull content from the target root. I fixed the missing grub static config in bootupd by manually pulling them when --src-root is passed, but the locale aren't present in the build root so they are not copied.

It would be nicer to have bootc actually chroot before calling bootupd, because it would have the added benefit to use the bootupd (and other) binaries that will ship on the image rather than what is in the buildroot.

Another way would be to use our image as the buildroot, but once again, we don't ship python.

[jbtrystram]

Ok I cleaned up the commits and addressed everything. I started a full test run with a COSA image containing this PR and the manifest config knob in the staging pipeline

[jbtrystram]

Ok I cleaned up the commits and addressed everything. I started a full test run with a COSA image containing this PR and the manifest config knob in the staging pipeline

a few tests are failing, i'll look into it tomorrow

[jbtrystram]

Next up is https://github.com/bootc-dev/bootc/issues/1710 ( we want to set sysroot.bls-append-except-default in the ostree repo config) and also /sysroot is not labeled correctly

[jbtrystram]

Fixed the labels issue in the last commit