bootc icon indicating copy to clipboard operation
bootc copied to clipboard

bootc switch --mutate-in-place failed on Fedora 42

Open henrywang opened this issue 1 year ago • 6 comments

Run bootc switch on fedora-bootc:42 system. Failed to run with --mutate-in-place.

[fedora@fedora ~]$ sudo bootc switch --mutate-in-place --transport registry quay.io/fedora/fedora-bootc:42
ERROR Switching: Writing origin: Read-only file system (os error 30)
[fedora@fedora ~]$ sudo bootc switch --transport registry quay.io/fedora/fedora-bootc:42
layers already present: 0; layers needed: 65 (820.2 MB)
Fetching layers ░░░░░░░░░░░░░░░░░░░░ 1/65 
 └ Fetching ░░░░░░░░░░░░░░░░░░░░ 1.13 MiB/63.27 MiB (34.73 MiB/s) ostree chunk 66763e94edad66cd05972

It's on Fedora 42 only.

henrywang avatar Dec 03 '24 04:12 henrywang

I get the same error when starting from a F41 host system too; to be clear this command was always only intended to be run from the Anaconda environment, where we aren't mounting the sysroot readonly today...or at least we weren't.

Are you sure this isn't specific to the ISO version? I will need to look at that

cgwalters avatar Dec 03 '24 17:12 cgwalters

Can't find this issue in my anaconda-iso test for fedora-bootc:41 image (https://artifacts.osci.redhat.com/testing-farm/e372b9f2-ecc9-487b-b331-5d883a1374ec/).

Are you sure this isn't specific to the ISO version? I will need to look at that

Do you mean the Volume id of ISO file? Thanks.

henrywang avatar Dec 03 '24 23:12 henrywang

I checked the install log, No any %POST error logs in console but bootc switch --mutate-in-place --transport registry 192.168.100.1:5000/bootc-workflow-test:ldgz does not work indeed.


Installing boot loader
.
Performing post-installation setup tasks
.
Importing certificates
.
Configuring installed system
.
Writing network configuration
.
Creating users
.
Configuring addons
.
Generating initramfs
....
Storing configuration files and kickstarts
.
Running post-installation scripts
...
Complete!

henrywang avatar Feb 15 '25 07:02 henrywang

Now this issue exists on fedora-bootc:42 and fedora-bootc:43/rawhide now.

henrywang avatar Feb 15 '25 07:02 henrywang

I gave this a quick test and I'm not able to reproduce it. I used a Fedora 42 installer ISO with quay.io/fedora/fedora-bootc:42. My test kickstart is quite simple:

$ cat inst.ks
# Basic setup
text
network --bootproto=dhcp --device=link --activate
# Basic partitioning
clearpart --all --initlabel --disklabel=gpt
reqpart --add-boot
part / --grow --fstype xfs

ostreecontainer --url quay.io/fedora/fedora-bootc:42

services --enabled=sshd
rootpw --iscrypted locked
sshkey --username root "<key>"
reboot

%post --erroronfail
bootc switch --mutate-in-place --transport registry quay.io/fedora/fedora-bootc:41
%end

After install I do see that the system origin is pointed back:

[root@fedora ~]# bootc status
● Booted image: quay.io/fedora/fedora-bootc:41
        Digest: sha256:f0fdf95cbd4809aeed3b305204483495ab3f62fd3395b77594b93057f7bf68dc (amd64)
       Version: 42.20250501.0 (2025-05-01T15:39:22Z)

Though I guess the next step here is to try using b-i-b to make a custom ISO, it could somehow be specific to that.

cgwalters avatar May 01 '25 18:05 cgwalters

I'm having this problem now, with f42:

my config.toml

[customizations.installer.kickstart]
contents = """
graphical

%post --erroronfail
bootc switch --mutate-in-place --transport registry quay.io/fedora/fedora-bootc:latest
%end

on the installed system

$ sudo bootc status
● Booted image: oci:/run/install/repo/container
        Digest: sha256:c3d57b3b5a539223c61faf3c2027e680a5ead678a126b60d8e95f1513f751507 (amd64)
       Version: 42.20250501.0 (2025-05-01T15:39:22Z)

jasonbrooks avatar May 01 '25 20:05 jasonbrooks

If anyone wants to chase debugging this I'd try adding a findmnt like

%post --erroronfail
findmnt
bootc switch --mutate-in-place --transport registry quay.io/fedora/fedora-bootc:latest
%end

But basically this needs someone (which may need to be me at some point) to log into the anaconda shell on failure and see what's going on.

cgwalters avatar Jun 02 '25 13:06 cgwalters

@cgwalters hopefully the following will help with troubleshooting:

I'm using bootc-image-builder and building a cayo:fedora ISO installer which is based off fedora:42.

sudo podman run \
    --rm \
    -it \
    --privileged \
    --pull=newer \
    --security-opt label=type:unconfined_t \
    -v ./config.toml:/config.toml:ro \
    -v ./output:/output \
    -v /var/lib/containers/storage:/var/lib/containers/storage \
    quay.io/centos-bootc/bootc-image-builder:latest \
    --type anaconda-iso \
    ghcr.io/ublue-os/cayo:fedora

I'm using the following config.toml as mentioned in the block above:

[customizations.installer.kickstart]
contents = """
text

%post --erroronfail --log=/root/bootc_switch.log
findmnt
bootc switch --mutate-in-place --transport registry ghcr.io/ublue-os/cayo:fedora
rm -f /var/lib/systemd/random-seed
%end

zerombr
clearpart --all --initlabel --disklabel=gpt
reqpart --add-boot
part / --grow --fstype=xfs --ondisk=sda
network --device=link --activate --onboot=True

timezone Pacific/Auckland
lang en_NZ

rootpw --iscrypted $y$....#HASH_OF_INITIAL_PASSWORD

#reboot
"""

[customizations.installer.modules]
enable = [
  "org.fedoraproject.Anaconda.Modules.Network",
  "org.fedoraproject.Anaconda.Modules.Payloads",
  "org.fedoraproject.Anaconda.Modules.Security",
  "org.fedoraproject.Anaconda.Modules.Services",
  "org.fedoraproject.Anaconda.Modules.Storage",
  "org.fedoraproject.Anaconda.Modules.Users",
  "org.fedoraproject.Anaconda.Modules.Localization",
  "org.fedoraproject.Anaconda.Modules.Timezone",
]
disable = [
  "org.fedoraproject.Anaconda.Modules.Subscription"
]

Note that I have commented out reboot so when it completes, it waits for me to press ENTER to reboot. It also means I can CTRL+B 2 to get into the terminal and look around. I've inserted the findmnt into the %post block as requested.

There is no bootc_switch.log in /root or /mnt/sysroot/var/roothome so it's like it's not running the %post block at all and as far as the installer is concerned, it's completed the install successfully. Switching to the various other terminals via CTRL+B number there's no findmnt output on any of them.

Also please note that on your successful test, you were using a remote registry as your initial source, whereas bootc-image-builder's anaconda-iso actually has the initial container image on the ISO itself. Therefore, the ostreecontainer directive is as follows in osbuild-base.ks on the ISO and the subsequent /mnt/sysroot/var/roothome/anaconda.cfg post-install:

ostreecontainer  --transport="oci" --url="/run/install/repo/container"

Post Reboot: Like others, the output of bootc status gives the following booted image rather than the switched to registry image: Booted image: oci:/run/install/repo/container

Not sure if this is anything that's noteworthy or not, but I also get this message in the console the first time I run bootc status: SELinux: Context unconfined_u:object_r:invalid_bootcinstall_testlabel_t:s0 is not valid (left unmapped).

If I run bootc status again it doesn't produce that SELinux message.

I'm happy to help troubleshoot to the best of my ability, with instruction from you (I'm relatively new to the Fedora / Bootc / Anaconda way of doing things, but not new to linux itself). Hopefully the above helps.

glen-gibson avatar Jul 16 '25 03:07 glen-gibson

Had some success today - pulled latest versions of bootc-image-builder and cayo:fedora, %post now runs bootc switch and generates a log file with the --log flag supplied, which it wasn't doing before.

Still getting the weird SELinux message though if I boot up and run bootc status for the first time - but it is referencing the correct url instead of the local OCI. Progress is progress.

glen-gibson avatar Jul 17 '25 04:07 glen-gibson

Might be the same issue: https://github.com/rhinstaller/anaconda/discussions/6397#discussioncomment-13790351

bam80 avatar Jul 17 '25 18:07 bam80

@cgwalters this might have narrowed things down: https://github.com/osbuild/bootc-image-builder/issues/968#issuecomment-3086099219

glen-gibson avatar Jul 18 '25 00:07 glen-gibson

Not sure if this is anything that's noteworthy or not, but I also get this message in the console the first time I run bootc status: SELinux: Context unconfined_u:object_r:invalid_bootcinstall_testlabel_t:s0 is not valid (left unmapped).

It's completely unrelated. I filed https://github.com/bootc-dev/bootc/issues/1434

cgwalters avatar Jul 21 '25 11:07 cgwalters