`cosign copy` fails when copying to Harbor

[atoy3731]

Description

I've validated that cosign copy works exactly as expected when copying from Azure ACR to registry v2 and Jfrog's JCR, however attempting to copy to 2 separate Harbor registries, I get an error (note - heavily redacted):

root@harbor-instance:/home# cosign copy acr.source.redacted/example/image:v1.0.0 harbor.destination.redacted/example/image:v1.0.0
Copying acr.source.redacted/example/image:sha256-6dbf9a9ce5c0370a72ce7b5b1671f95744320e4ff37c8bdf9aea09c92abcdefg.sig to harbor.destination.redacted/example/image:sha256-6dbf9a9ce5c0370a72ce7b5b1671f95744320e4ff37c8bdf9aea09c92abcdefg.sig...
Error: PUT https://harbor.destination.redacted/v2/example/image/manifests/sha256-6dbf9a9ce5c0370a72ce7b5b1671f95744320e4ff37c8bdf9aea09c92abcdefg.sig: NOT_FOUND: artifact example/image@sha256:6dbf9a9ce5c0370a72ce7b5b1671f95744320e4ff37c8bdf9aea09c92abcdefg not found
main.go:62: error during command execution: PUT https://harbor.destination.redacted//v2/example/image/manifests/sha256-6dbf9a9ce5c0370a72ce7b5b1671f95744320e4ff37c8bdf9aea09c92abcdefg.sig: NOT_FOUND: artifact example/image@sha256:6dbf9a9ce5c0370a72ce7b5b1671f95744320e4ff37c8bdf9aea09c92abcdefg not found

Just for awareness, I do have the example project created in my Harbor instance and have validated that I can push/pull images to/from it using docker.

I have also validated that cosign sign against the Harbor registry does work as expect. So this only seems to be affecting the cosign copy command from what I've seen.

Version

Cosign:

GitVersion:    v1.11.1
GitCommit:     b3b6ae25362dc2c92c78abf2370ba0342ee86b2f
GitTreeState:  clean
BuildDate:     2022-08-24T15:09:32Z
GoVersion:     go1.18.5
Compiler:      gc
Platform:      linux/amd64

Harbor (latest available chart, running in K3s):

NAME  	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART       	APP VERSION
harbor	harbor   	4       	2022-08-30 00:38:43.980448592 +0000 UTC	deployed	harbor-1.9.3	2.5.3

[znewman01]

@jdolitsky can I ask for your OCI registry quirks expertise on this?

[ChristianCiach]

It looks like cosign tries to copy the signature first, before copying the actual image. Harbor has special support for cosign signatures and it seems that harbor only accepts signature images for images that already exists in the registry.

Running cosign copy --verbose indeed shows that the .sig artifact is copied first. Harbor than complains that there is no image that the tag of the .sig image references:

2022/10/05 17:20:39 --> PUT https://my-registry/v2/my-image-repo/manifests/sha256-a0cfb676dc0e6b4111cde288fbe91192662c47b6714d44de38648f84bbcf9184.sig
2022/10/05 17:20:39 PUT /v2/my-image-repo/manifests/sha256-a0cfb676dc0e6b4111cde288fbe91192662c47b6714d44de38648f84bbcf9184.sig HTTP/1.1
Host: my-registry
User-Agent: cosign/v1.12.1 (linux; amd64) go-containerregistry/v0.11.0
Content-Length: 558
Authorization: <redacted>
Content-Type: application/vnd.oci.image.manifest.v1+json
Accept-Encoding: gzip

{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","config":{"mediaType":"application/vnd.oci.image.config.v1+json","size":243,"digest":"sha256:5e082e17544cb7c4d131e33d08ee495b656c56bff125e2d80a27c6a410f2498b"},"layers":[{"mediaType":"application/vnd.dev.cosign.simplesigning.v1+json","size":298,"digest":"sha256:114679d4239d7b9c8d7d670dd0dd27affbc67d3fb4f5a239f9e96e6199fb1986","annotations":{"dev.cosignproject.cosign/signature":"MEQCIHWk8aKBPMacUA+2S8NxI/BVKzr58ZQ7fxeUOO8hsj8ZAiAORZiz06PpxVB5j0mzeZnA54sq646a2oCmWzEczCIq1Q=="}}]}
2022/10/05 17:20:39 <-- 404 https://my-registry/v2/my-image-repo/manifests/sha256-a0cfb676dc0e6b4111cde288fbe91192662c47b6714d44de38648f84bbcf9184.sig (34.875321ms)
2022/10/05 17:20:39 HTTP/1.1 404 Not Found
Content-Length: 159
Content-Type: application/json; charset=utf-8
Date: Wed, 05 Oct 2022 15:20:39 GMT
Server: nginx
Set-Cookie: sid=531255556ef139fad7fed85a425231f3; Path=/; HttpOnly
Strict-Transport-Security: max-age=15768000
X-Request-Id: 7084fc04-c15b-41c7-ba7c-68c5a7ba0e9f

{"errors":[{"code":"NOT_FOUND","message":"artifact my-image-repo@sha256:a0cfb676dc0e6b4111cde288fbe91192662c47b6714d44de38648f84bbcf9184 not found"}]}

As you can see, cosign tries to PUT the artifact with the tag sha256-a0cfb676dc0e6b4111cde288fbe91192662c47b6714d44de38648f84bbcf9184.sig and then Harbor complains that there is no artifact with the digest sha256:a0cfb676dc0e6b4111cde288fbe91192662c47b6714d44de38648f84bbcf9184.

It would probably work if cosign copies the image first before copying the signature and other related artifacts.

[ChristianCiach]

This also means that you probably cannot use the COSIGN_REPOSITORY environment variable to store only the signature in Harbor. Harbor does not accept signatures without the corresponding image. I would argue that this is probably a bug in Harbor.

[ChristianCiach]

@developer-guy Since you implemented the current order of copying the images, I think it makes sense to at least notify you about that.

[VenutNSA]

Hello all, has there been a solution to this problem? I have tried getting around this by using "crane copy", however only the image is copied, without the signature

[zackbradys]

Hey @atoy3731, you may want to try this again. Appears to be working now!