containers
Don't look for host policy with `--source-imageref`
This is like https://github.com/bootc-dev/bootc/issues/1410 but for SELinux. Basically we shouldn't look at the host system policy https://github.com/bootc-dev/bootc/blob/03fa72b194070f6973451aeb52279957535567b0/crates/lib/src/install.rs#L631 until later, and then reconcile the labels on files we wrote using that.
The workaround for this is to just install a policy in the buildroot (e.g. selinux-policy-targeted in Fedora).
Hi @cgwalters , thanks for the update. I have tried the workaround and it is more complicated.
The 1st thing is that standalone package install is not enough as SELinux removal is part of the Lorax runtime removal template: https://github.com/weldr/lorax/blob/913cb696fd992e867f6e6ea043293da6dad7ba6e/share/templates.d/99-generic/runtime-cleanup.tmpl#L21
To have it working I have applied custom Lorax template and sideloaded the selinux-policy-targeted rpm package. End result was OS not booting at all:
Thanks to the team debugging effort I was able to boot the image by manually adding selinux=0 to kernel parameters list in grub.
This solved the bootc SELinux policies complain but I reached the next error:
error: Installing to filesystem: Creating ostree deployment: Creating importer: failed to invoke method OpenImage: failed to invoke method OpenImage: 'overlay' is not supported over overlayfs, a mount_program is required: backing file system is unsupported for this graph driver
I am not sure if this is some authentication to registry issue (https://access.redhat.com/solutions/7086221) or yet another blocker we need to resolve.
Thanks to the team debugging effort I was able to boot the image by manually adding selinux=0 to kernel parameters list in grub.
Hmm...that will have other side effects. Doesn't Anaconda run today in permissive=1? A bit odd that including the policy package would cause this. I guess this does need some digging.
overlay' is not supported over overlayfs,
This is a different issue, it's not related to authentication to the registry. The problem is because Anaconda is mounting an overlayfs for /. Try just typing e.g. podman images - it fails with the same error.
Though Fedora Workstation is set up the same way, but doesn't fail. I am not sure why. It's very clear that /var/lib/containers/storage is an overlay there too which should fail but doesn't...I am not seeing any obvious changes to the podman root storage config to redirect it set up in Workstation.
It's notable though the "LiveOS" approach used for Fedora CoreOS is different because we don't default to writability for / so we make dedicated tmpfs for /etc and /var which avoids overlay-on-overlay consistently.
Try just typing e.g. podman images - it fails with the same error.
So a workaround here is mount -t tmpfs tmpfs /var/lib/containers e.g. (probably better done via a systemd .mount unit).
But
It's notable though the "LiveOS" approach used for Fedora CoreOS
To expand on this the really nice thing about the design we have for image-based derivatives is that we have a strong stance that the system should support an empty /var so we can sanely just make a tmpfs for all of var, and that's that.
Just for the record - I have tired this with various combinations of selinux=0/1 and enforcing=0/1. So far result is the same.
Hi @cgwalters , I have an update about the testing:
[anaconda root@fedora ~]# bootc install to-filesystem --stateroot=default --source-imgref=containers-storage:quay.io/centos-bootc/centos-bootc:stream9 --target-imgref=containers-storage:quay.io/centos-bootc/centos-bootc:stream9 /mnt/sysimage
Installing image: docker://containers-storage:quay.io/centos-bootc/centos-bootc:stream9
Initializing ostree layout
error: Installing to filesystem: Creating ostree deployment: Creating imgstorage: Initializing images: No such file or directory (os error 2)
This is with mount -t tmpfs tmpfs /var/lib/containers:
tmpfs on /var/lib/containers type tmpfs (rw,relatime,inode64)
I have checked content of the /var/lib/containers and tried variants with:
- empty dir
- copy of premounted content
The result is the same.
My other observation is that with mount in place bootc fails faster.
All the above has been tested with selinux=0 kernel parameter.
OK, it looks like it may be caused by small amount of RAM on my VM. After changing from 4G to 20G I reached new point:
error: Installing to filesystem: Creating ostree deployment: Creating importer: failed to invoke method OpenImage: failed to invoke method OpenImage: reference "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.imagestore=/usr/lib/containers/storage,overlay.mountopt=nodev,metacopy=on]quay.io/centos-bootc/centos-bootc:stream9" does not resolve to an image ID
This is starting to show that we need to store the container image on the disk instead of memory I would say. The RAM requirement might be insane...
56 pub enum Transport {
57 /// A remote Docker/OCI registry (`registry:` or `docker://`)
58 Registry,
59 /// A local OCI directory (`oci:`)
60 OciDir,
61 /// A local OCI archive tarball (`oci-archive:`)
62 OciArchive,
63 /// A local Docker archive tarball (`docker-archive:`)
64 DockerArchive,
65 /// Local container storage (`containers-storage:`)
66 ContainerStorage,
67 /// Local directory (`dir:`)
68 Dir,
69 }
Now I realized I was using wrong repo modifier. Now with this call:
bootc install to-filesystem --stateroot=default --source-imgref=registry:quay.io/fedora-testing/fedora-bootc:rawhide-minimal --target-imgref=registry:quay.io/fedora-testing/fedora-bootc:rawhide-minimal /mnt/sysimage
@cgwalters one question here - bootc expects to have an empty target directory, just to complain later there is no /boot directory presented. Can you elaborate a bit about bootc using the /boot directory? I guess it expects to find a kernel there together with the grub config? I tried with empty /boot directory on target partition and it ends up with:
[anaconda root@fedora ~]# bootc install to-filesystem --stateroot=default --source-imgref=registry:quay.io/fedora-testing/fedora-bootc:rawhide-minimal --target-imgref=registry:quay.io/fedora-testing/fedora-bootc:rawhide-minimal /mnt/sysimage
Installing image: docker://registry:quay.io/fedora-testing/fedora-bootc:rawhide-minimal
Initializing ostree layout
layers already present: 0; layers needed: 1 (438.5 MB)
Fetched layers: 418.23 MiB in 55 seconds (7.65 MiB/s) Deploying container image: done (9 seconds) Running bootupctl to install bootloader
> bootupctl backend install --write-uuid --update-firmware --auto --device /dev/vda /mnt/sysimage
error: Installing to filesystem: Installing bootloader: No such file or directory (os error 2)
We are almost there :)
With the 20G of RAM attached to VM the steep with mount -t tmpfs tmpfs /var/lib/containers is not needed.
I have reached a new blocker related to bootloader deployment:
[anaconda root@fedora ~]# bootc install to-filesystem --stateroot=default --source-imgref=registry:quay.io/fedora-testing/fedora-bootc:rawhide-minimal --target-imgref=registry:quay.io/fedora-testing/fedora-bootc:rawhide-minimal /mnt/sysimage
Installing image: docker://registry:quay.io/fedora-testing/fedora-bootc:rawhide-minimal
Initializing ostree layout
layers already present: 0; layers needed: 1 (438.5 MB)
Fetched layers: 418.23 MiB in 54 seconds (7.78 MiB/s) Deploying container image: done (9 seconds) Running bootupctl to install bootloader
> bootupctl backend install --write-uuid --update-firmware --auto --device /dev/vda /mnt/sysimage
error: Installing to filesystem: Installing bootloader: No such file or directory (os error 2)
It looks like it is the issue mentioned by @jbtrystram here: https://github.com/bootc-dev/bootc/issues/1455 What would be a valid workaround around this?
It looks like it is the issue mentioned by @jbtrystram here: https://github.com/bootc-dev/bootc/issues/1455 What would be a valid workaround around this?
You can try with a build that includes https://github.com/bootc-dev/bootc/pull/1460 it fixed my issue.
Also, try to set BOOTC_BOOTLOADER_DEBUG=1 to learn more about what is missing
@jbtrystram - thanks for the fast answer:) If I can trick bootupctl using some symlinks / copy pasting files in some expected location? I will use BOOTC_BOOTLOADER_DEBUG=1 for sure too.
If I can trick bootupctl using some symlinks / copy pasting files in some expected location?
You can try with a build that includes https://github.com/bootc-dev/bootc/pull/1460 it fixed my issue.
Yes easiest is probably to to install a build from https://copr.fedorainfracloud.org/coprs/rhcontainerbot/bootc/ in your testing environment via e.g. dnf copr enable or just directly rpm -Uvh https://download.copr.fedorainfracloud.org/results/rhcontainerbot/bootc/fedora-42-aarch64/09356593-bootc/bootc-202507301726.g689340333d-1.fc42.aarch64.rpm or whatever
This is starting to show that we need to store the container image on the disk instead of memory I would say. The RAM requirement might be insane...
In this model we're still fetching the image from a registry and directly writing it to the target filesystem, right?
If I can trick bootupctl using some symlinks / copy pasting files in some expected location?
Or just install a build from https://copr.fedorainfracloud.org/coprs/rhcontainerbot/bootc/ in your testing environment?
@cgwalters those are the build of latest master branch? I see lots of State: failed - will this not interfere with our testing on Anaconda env somewhere else?
@cgwalters those are the build of latest master branch?
Yes it auto-builds from git main
I see lots of State: failed - will this not interfere with our testing on Anaconda env somewhere else?
When one digs in note that it's only rhel and opensuse builds that are failing. The RHEL ones are because its Rust is too old. (centos stream is new enough)
After adding missing bootupd package for the first time the install succeed \o/
[anaconda root@fedora ~]# bootc install to-filesystem --stateroot=default --source-imgref=registry:quay.io/fedora-testing/fedora-bootc:rawhide-minimal --target-imgref=registry:quay.io/fedora-testing/fedora-bootc:rawhide-minimal /mnt/sysimage
Installing image: docker://registry:quay.io/fedora-testing/fedora-bootc:rawhide-minimal
Initializing ostree layout
layers already present: 0; layers needed: 1 (438.5 MB)
Fetched layers: 418.23 MiB in 22 seconds (18.69 MiB/s) Deploying container image: done (9 seconds) Running bootupctl to install bootloader
> bootupctl backend install --write-uuid --update-firmware --auto --src-root /mnt/sysimage/ostree/deploy/default/deploy/e68ecc5647787a93c45eca99a8731e9edad51b1f7c605b04a33a089b32c9ca3d.0 --device /dev/vda /mnt/sysimage
Added 01_users.cfg
Added 10_blscfg.cfg
Added 14_menu_show_once.cfg
Added 30_uefi-firmware.cfg
Added 41_custom.cfg
Installed: grub.cfg
Installed: bootuuid.cfg
Trimming root
.: 17.9 GiB (19205865472 bytes) trimmed
Finalizing filesystem root
Installation complete!