'apptainer build' fails when using signed Docker-style Containers

[e4t]

Version of Apptainer

$ apptainer.test --version
apptainer version 1.2.5-0

Expected behavior

Using the following definition:

Bootstrap: docker
From: registry.suse.com/bci/bci-base:15.5

%appinstall project
    zypper -n install -y ca-certificates
    zypper clean --all

apptainer build slebci.sif slebci.dev should successfully build a SIF container using SLE BCI.

Actual behavior

Instead apptainer is unable to handle signed docker images like those provided by the SUSE registry. apptainer -d build slebci.sif slebci.dev fails and ends with the following output:

[...]
INFO    [U=0,P=1]          Full()                        Starting build...
DEBUG   [U=0,P=1]          Get()                         Reference: registry.suse.com/bci/bci-base:15.5
DEBUG   [U=0,P=1]          getDockerRefDigest()          docker.GetDigest source image digest for docker://registry.suse.com/bci/bci-base:15.5 is 55fc283fa0574b7336428305530b445fc39fe1ee7dd47aec167af6bbe7ce82f6
DEBUG   [U=0,P=1]          getDockerRefDigest()          docker.GetDigest digest for docker://registry.suse.com/bci/bci-base:15.5 is a70d1db70ce1fe6e8ad9f009c03f6f4a6d1b930201780f5e09c509caa132aa8e
DEBUG   [U=0,P=1]          getRefDigest()                GetManifest digest for docker://registry.suse.com/bci/bci-base:15.5 is a70d1db70ce1fe6e8ad9f009c03f6f4a6d1b930201780f5e09c509caa132aa8e
Getting image source signatures
Checking if image destination supports signatures
DEBUG   [U=0,P=1]          cleanUp()                     Cleaning up "/tmp/build-temp-3912847688/rootfs" and "/tmp/bundle-temp-2163527452"
FATAL   [U=0,P=1]          runBuildLocal()               While performing build: conveyor failed to get: while fetching image: initializing source oci:/home/auser/.apptainer/cache/blob:a70d1db70ce1fe6e8ad9f009c03f6f4a6d1b930201780f5e09c509caa132aa8e: copying system image from manifest list: Can not copy signatures to oci:/home/auser/.apptainer/cache/blob:a70d1db70ce1fe6e8ad9f009c03f6f4a6d1b930201780f5e09c509caa132aa8e: Pushing signatures for OCI images is not supported
DEBUG   [U=0,P=5193]       func1()                       Wait for process 5213 complete with status 65280, error <nil>
DEBUG   [U=0,P=5193]       Master()                      Child exited with exit status 255

Steps to reproduce this behavior

See above How can others reproduce this issue/problem?

What OS/distro are you running

$ cat /etc/os-release
NAME="SLES"
VERSION="15-SP6"
VERSION_ID="15.6"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP6"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp6"
DOCUMENTATION_URL="https://documentation.suse.com/"

How did you install Apptainer

zypper -n in -y apptainer

Analysis

This issue is unrelated to issue #1034. The solution proposed there will not solve this problem. The failure is more serious and it was likely introduced with commit 68d704026 to github.com/containers/image when support for cosign (sigstore) attachments was added: Image destinations handled by github.com/containers/image/v5/oci/layout are unable to handle (store) signatures as the error message suggests. Instead of ignoring the signature and continuing it causes a failure. The call chain that leads up to this is:

conveyorPacker_oci.go:Get() ->  internal/pkg/build/sources/conveyorPacker_oci.go:fetch()
-> github.com/containers/image/v5/copy/copy.go: copy.Image()
 -> github.com/containers/image/v5/copy/single.go: copySingleImage 
  -> github.com/containers/image/v5/copy/sign.go: c.sourceSignatures() 
   -> github.com/containers/image/v5/oci/archive/oci_dest.go: c.dest.SupportsSignatures(ctx)

[mslacken]

I think its the same problem with oci container library and was solved in warewulf with https://github.com/warewulf/warewulf/commit/534ae70ca408ba3bc7496899305ef05b84250791

[GodloveD]

Sorry to parade my ignorance here, but is there any way that we (and maybe WW also) could make this work by actually checking and verifying the signature?

[mslacken]

@GodloveD The error here is that the target to which the layers are written can't hold the signatures. I did not find an easy way to check the signatures, but if you know one this would be great. I also can't reproduce this error on non openSUSE leap systems, so this seems to be related either to the

• go version
• target filesystem

[DrDaveD]

@e4t Does your SUSE host system contain configuration in /etc/containers, something like what is referred to in this article?

[e4t]

I think its the same problem with oci container library and was solved in warewulf with warewulf/warewulf@534ae70

Yes, this is exactly what we need - but it is required in two places. Had I realized this earlier it would have saved me a day of digging around.

[e4t]

@GodloveD The error here is that the target to which the layers are written can't hold the signatures. I did not find an easy way to check the signatures, but if you know one this would be great. I also can't reproduce this error on non openSUSE leap systems, so this seems to be related either to the

Most of this is not handled by apptainer itself by by github.com/containers/images/v5. The error message is generated there as well. The code could be rewritten to use a different infrastructure like ggcr - as singularity has done. I have no idea if this handles signatures, though.

[e4t]

@DrDaveD -

@e4t Does your SUSE host system contain configuration in /etc/containers, something like what is referred to in this article?

Good question, I found this article.

It would not make a difference for this problem, though, as the way the container layers are stored for apptainer build .. does not allow for storing signatures and thus no checking will be done anyway.

[e4t]

Ok, ... learning while I'm trudging along - stumbling over the source code in github.com/containers/image/v5/docker/registries_d.go ... The handling of detached signatures can be configured the problem happens because of a pre-configuration on a SUSE SLE system: /etc/containers/registries.d/default.yaml has this:

docker:
  registry.suse.com:
    use-sigstore-attachments: true
  registry.suse.de:
    use-sigstore-attachments: true

Changing this to fix the problem should be considered as a workaround only, though.

[DrDaveD]

Ok so this shows how to reproduce the problem on any host operating system.

the way the container layers are stored for apptainer build .. does not allow for storing signatures

How do you know that? Can it be changed?

[e4t]

How do you know that? Can it be changed?

Well, the error message say so: "Pushing signatures for OCI images is not supported". This comes straight from github.org/image/v5/oci/layout/oci_dest.go. Therefore, I don't think it can be changed.

[DrDaveD]

That's got to be something that can be overridden in the API.

[DrDaveD]

Btw I expect that apptainer-1.4 won't use much containers/image code at all, because singularity-ce has converted their code to instead use go-containerregistry and I plan to do the same for apptainer. We won't be using the copy package at all.

[e4t]

Btw I expect that apptainer-1.4 won't use much containers/image code at all, because singularity-ce has converted their code to instead use go-containerregistry and I plan to do the same for apptainer. We won't be using the copy package at all.

Indeed. I had tested it with a recent singularity-ce and not encountered any issues with signatures. My patch was not meant as an ultimate solution but to make things work again with the current code. I'd expect it to fail with UBI on a RHEL as it has a similar setup for registry.access.redhat.com as we have for registry.suse.com (not sure about RHEL clones).

[DrDaveD]

This was discussed in today's community call. Since the library is going to be replaced anyway, we're OK with disabling the signature checks.

[e4t]

This was discussed in today's community call. Since the library is going to be replaced anyway, we're OK with disabling the signature checks.

As far as I can gather from the code, there are no signature checks for OCI images as OCI images do not support storing a signature. Currently, it only works if the registry does not provide sigstore attachments or the target system does not request to use them for this registry - in which case they are dropped regardless of the patch. So this patch doesn't disable anything that has been there before. This is different for OCI archives which support storing signatures but these are not used in the context where we ask to drop the signatures. It is certainly better to switch to a different destination format which allows to properly store and verify signatures.