fedora-43 tests start failing

[mvo5]

The TestCaseFedora43 tests start failing:

...
org.osbuild.bootc.install-to-filesystem: 19bb778fae4541936924e98952fc101eabf7f1782856dd0447ae1fef4ad3ac61 {
  "kernel-args": [
    "rw",
    "console=tty0",
    "console=ttyS0",
    "systemd.journald.forward_to_console=1"
  ],
  "target-imgref": "quay.io/fedora/fedora-bootc:43"
}
device/disk (org.osbuild.loopback): loop0 acquired (locked: False)
mount/- (org.osbuild.btrfs): mounting /dev/loop0p4 -> /store/tmp/buildroot-tmp-e004ml_u/mounts/
mount/boot (org.osbuild.xfs): mounting /dev/loop0p3 -> /store/tmp/buildroot-tmp-e004ml_u/mounts/boot
mount/boot-efi (org.osbuild.fat): mounting /dev/loop0p2 -> /store/tmp/buildroot-tmp-e004ml_u/mounts/boot/efi
Mount transient overlayfs for /etc/containers
Creating bind mount for run/osbuild/containers
Installing image: docker://quay.io/fedora/fedora-bootc:43
Initializing ostree layout
ERROR Installing to filesystem: Creating ostree deployment: invalid reference format
Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 53, in <module>
    r = main(args["options"], args["inputs"], args["paths"])
  File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 48, in main
    subprocess.run(pargs, env=env, check=True)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['bootc', 'install', 'to-filesystem', '--source-imgref', 'containers-storage:[overlay@/run/osbuild/containers/storage+/run/containers/storage]756a5b3dae6f3ebe28f8bc363c0b769ae76b6e6d2d3217bed542b20c68db3359', '--skip-fetch-check', '--generic-image', '--karg', 'rw', '--karg', 'console=tty0', '--karg', 'console=ttyS0', '--karg', 'systemd.journald.forward_to_console=1', '--target-imgref', 'quay.io/fedora/fedora-bootc:43', '/run/osbuild/mounts']' returned non-zero exit status 1.
mount/boot-efi (org.osbuild.fat): umount: /store/tmp/buildroot-tmp-1l1wkyzi/mounts/boot/efi unmounted
mount/boot (org.osbuild.xfs): umount: /store/tmp/buildroot-tmp-1l1wkyzi/mounts/boot unmounted
mount/- (org.osbuild.btrfs): umount: /store/tmp/buildroot-tmp-1l1wkyzi/mounts/ unmounted

⏱  Duration: 1s
manifest - failed
Failed

Fwiw, it looks like the error is coming from https://github.com/containers/image/blob/main/docker/reference/reference.go#L41 so maybe indirectly via ImageProxyConfig (a bit of a guess, I need to dig a bit more into bootc/ostree)

I wonder if our `containers-storage:[overlay@/run/osbuild/containers/storage+/run/containers/storage]756a5b3dae6f3ebe28f8bc363c0b769ae76b6e6d2d3217bed542b20c68db3359' is breaking and the new container lib cannot parse it anymore…?

As a workaround https://github.com/osbuild/bootc-image-builder/pull/933 is opened

[mvo5]

Colin pointed out this might be related to https://github.com/bootc-dev/bootc/pull/1302 - and indeed adding a testcase that is close to what we pass to bootc yield the same error:

$ git describe --always
v1.2.0-22-gcbbd345

$ git diff
diff --git a/lib/src/spec.rs b/lib/src/spec.rs
index f87f242..e087487 100644
--- a/lib/src/spec.rs
+++ b/lib/src/spec.rs
@@ -401,6 +401,11 @@ mod tests {
                 "/tmp/image-dir".to_string(),
                 "dir",
             ),
+           (
+               format!("[overlay@/run/osbuild/containers/storage+/run/containers/storage]{}", sample_digest),
+               format!("[overlay@/run/osbuild/containers/storage+/run/containers/storage]{}", sample_digest),
+               "containers-storage",
+           ),
         ];
 
         for (initial, expected, transport) in test_cases {

$ cargo test test_image_reference_canonicalize
    Finished `test` profile [optimized + debuginfo] target(s) in 0.20s
     Running unittests src/main.rs (target/debug/deps/bootc-2c7d51933da9f1e0)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running unittests src/blockdev.rs (target/debug/deps/bootc_blockdev-2e091aebb9788916)

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.00s

     Running unittests src/lib.rs (target/debug/deps/bootc_lib-541d99303231c0b5)

running 1 test
test spec::tests::test_image_reference_canonicalize ... FAILED

failures:

---- spec::tests::test_image_reference_canonicalize stdout ----
thread 'spec::tests::test_image_reference_canonicalize' panicked at lib/src/spec.rs:420:17:
Failed to canonicalize [overlay@/run/osbuild/containers/storage+/run/containers/storage]sha256:5db6d8b5f34d3cbdaa1e82ed0152a5ac980076d19317d4269db149cbde057bb2 with transport containers-storage: invalid reference format
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    spec::tests::test_image_reference_canonicalize

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 69 filtered out; finished in 0.03s

error: test failed, to rerun pass `-p bootc-lib --lib`

[alexlarsson]

Yeah, the problem is that containers-storage: transport reference can contain options. Something like this seems to fix it:

diff --git a/ostree-ext/src/container/mod.rs b/ostree-ext/src/container/mod.rs
index 5b12a432..9d15e6ca 100644
--- a/ostree-ext/src/container/mod.rs
+++ b/ostree-ext/src/container/mod.rs
@@ -156,6 +156,15 @@ impl TryFrom<&str> for ImageReference {
                 .strip_prefix("//")
                 .ok_or_else(|| anyhow!("Missing // in docker:// in {}", value))?;
         }
+        if transport_name == "containers-storage" {
+            // containters-storage transports allow specifying options
+            // in brackets before the image, strip them out
+            if name.starts_with('[') {
+                if let Some(end) = name.find(']') {
+                    name = &name[end+1..]
+                }
+            }
+        }
         Ok(Self {
             transport,
             name: name.to_string(),

Although I'm not 100% sure that is the right solution.

[cgwalters]

This is tracked by https://github.com/bootc-dev/bootc/issues/1308 and is fixed