Signature not found in GCP Artifact Registry

[david-freistrom]

What is not working as expected?

After notation sign a container image, GCP artifact registry shows the signature as an OCI artifact in GCP artifact-registry beside the docker image but no attachments are defined and verify or ls cannot find attached signature.

No attachments on the image nor on the signature OCI artifacts But the Artifacts exist and refers to the image in the manifest subject.

notation sign --signature-format jws --id personal/github/all --plugin hc-vault --plugin-config=kvName=kubeflow --plugin-config=transitKeyName=image-signing europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47

Successfully signed europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47

 notation ls europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47

europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 has no associated signature

notation verify europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47

Error: signature verification failed: no signature is associated with "europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47", make sure the artifact was signed successfully

cat /home/david/.config/notation/trustpolicy.json

{
    "version": "1.0",
    "trustPolicies": [
        {
            "name": "example-images",
            "registryScopes": [ "*" ],
            "signatureVerification": {
                "level" : "strict"
            },
            "trustStores": [ "ca:example.com" ],
            "trustedIdentities": [
                "*"
            ]
        }
    ]
}

What did you expect to happen?

notation verify and ls shows me a valid signature attached to the image

How can we reproduce it?

Build and Push an docker image to artifact registry in GCP sign it using notation list or verify it using notation

Describe your environment

Linux Ubuntu 24.04.2 LTS

5.15.167.4-microsoft-standard-WSL2 #1 SMP Tue Nov 5 00:21:55 UTC 2024

Notation - a tool to sign and verify artifacts. Version: 1.3.1 Go version: go1.23.6 Git commit: bb571ddfb04f9d175daf64520fd18e7bea8ecbdf

What is the version of your Notation CLI or Notation Library?

Notation - a tool to sign and verify artifacts.

Version: 1.3.1 Go version: go1.23.6 Git commit: bb571ddfb04f9d175daf64520fd18e7bea8ecbdf

[FeynmanZhou]

Hi @david-freistrom ,

In Notation v1.3.1, the notation sign command defaults to storing signatures using the OCI referrers tag schema for maximum compatibility. The notation verify/list/inspect commands will attempt to use the referrers API first and automatically fall back to the referrers tag schema if the referrers API is not supported by the registry. See this blog post for context.

It looks like the fallback is not executed as expected. Can you please try to add a flag --allow-referrers-api false to your notation list and notation verify commands as follows? This is an experimental flag in v1.x.

export NOTATION_EXPERIMENTAL=1

notation ls europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 --allow-referrers-api false

notation verify europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 --allow-referrers-api false

[ghost]

@david-freistrom After reading through your issue, one possible cause is that GCP Artifact Registry is not fully OCI 1.1 compliant.

[david-freistrom]

@Two-Hearts Small correction: Attachments are present at the origin image. Didn't saw it in the first. But Notation doesn't work as expected for verify and ls

Regarding GCP Artifact Registry announcements, OCI 1.1 support was generally available at Oct. 03 2024.

[david-freistrom]

I am wondering why notation creates a second artifact with the digest as a tag beside the one with semantic version as a tag. The attachments are only added to the semantic versioned artifact but verify looks only for the digest tagged artifact. (It doesn't matter if I use semantic versioned tag or the digest for verify)

notation verify europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 -v

INFO Allowed to access the referrers API, fallback if not supported

INFO Reference sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 resolved to manifest descriptor: {MediaType:application/vnd.docker.distribution.manifest.v2+json Digest:sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 Size:1159 URLs:[] Annotations:map[] Data:[] Platform:<nil> ArtifactType:}

INFO Checking whether signature verification should be skipped or not

INFO Trust policy configuration: &{Name:corporate-data-analytics-images RegistryScopes:[*] SignatureVerification:{VerificationLevel:strict Override:map[] VerifyTimestamp:} TrustStores:[ca:example.org] TrustedIdentities:[*]}
INFO Check over. The signature verification level is not set to 'skip' in the trust policy.

Error: signature verification failed: no signature is associated with "europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47", make sure the artifact was signed successfully

[david-freistrom]

Verify works with notation 1.1.2. Looks like a bug in 1.3.1.

notation-1.1.2 verify europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc:1.0 -v

INFO Using the referrers tag schema

INFO Reference 1.0 resolved to manifest descriptor: {MediaType:application/vnd.docker.distribution.manifest.v2+json Digest:sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 Size:1159 URLs:[] Annotations:map[] Data:[] Platform:<nil> ArtifactType:}
Warning: Always verify the artifact using digest(@sha256:...) rather than a tag(:1.0) because resolved digest may not point to the same signed artifact, as tags are mutable.

INFO Checking whether signature verification should be skipped or not

INFO Trust policy configuration: &{Name:corporate-data-analytics-images RegistryScopes:[*] SignatureVerification:{VerificationLevel:strict Override:map[]} TrustStores:[ca:example.org] TrustedIdentities:[*]}

INFO Check over. Trust policy is not configured to skip signature verification

INFO Processing signature with manifest mediaType: application/vnd.oci.image.manifest.v1+json and digest: sha256:fc3d42c24a1742103ea57b687c990feec3eea2278bcece62a79a590018bb23d9

INFO Trust policy configuration: &{Name:corporate-data-analytics-images RegistryScopes:[*] SignatureVerification:{VerificationLevel:strict Override:map[]} TrustStores:[ca:example.org] TrustedIdentities:[*]}
Successfully verified signature for europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47

[david-freistrom]

@FeynmanZhou

export NOTATION_EXPERIMENTAL=1

Still not works. But notation 1.1.2 works as expected without your suggested settings.

[ghost]

@FeynmanZhou

export NOTATION_EXPERIMENTAL=1

Still not works. But notation 1.1.2 works as expected without your suggested settings.

@david-freistrom The above makes the problem even more likely to be a GCP side issue. Because with Notation 1.1.2, when sign/verify without export NOTATION_EXPERIMENTAL=1, Notation will always use referrers tag schema (OCI 1.0) instead of trying referrers api first, if not supported, then fallback to tag schema (OCI 1.1). Since GCP is OCI 1.0 compliant, everything works fine.

However, when using Notation v1.3.1, the verify/list/inspect commands automatically tries the referrers api first (OCI 1.1), your error starts to pop up. Because GCP does not fully support the try referrers api first then fall back to referrers tag schema (OCI 1.1) mechanism. If they do fully support it, the fall back should work, and the verification workflow eventually works the same as Notation v1.1.2.

(Note: Notation v1.3.1 has been tested with other OCI 1.1 compliant registries, a fully OCI 1.1 compliant registry won't return this error).

[ghost]

I am wondering why notation creates a second artifact with the digest as a tag beside the one with semantic version as a tag. The attachments are only added to the semantic versioned artifact but verify looks only for the digest tagged artifact. (It doesn't matter if I use semantic versioned tag or the digest for verify)

notation verify europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 -v

INFO Allowed to access the referrers API, fallback if not supported

INFO Reference sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 resolved to manifest descriptor: {MediaType:application/vnd.docker.distribution.manifest.v2+json Digest:sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47 Size:1159 URLs:[] Annotations:map[] Data:[] Platform: ArtifactType:}

INFO Checking whether signature verification should be skipped or not

INFO Trust policy configuration: &{Name:corporate-data-analytics-images RegistryScopes:[] SignatureVerification:{VerificationLevel:strict Override:map[] VerifyTimestamp:} TrustStores:[ca:example.org] TrustedIdentities:[]} INFO Check over. The signature verification level is not set to 'skip' in the trust policy.

Error: signature verification failed: no signature is associated with "europe-west3-docker.pkg.dev/xxx/personal-poc-ssdlc/dna-ssdlc-poc@sha256:dc07d0fa1972f91ddc09cfc31564429278eb2055f847acf414b1ed4501266c47", make sure the artifact was signed successfully

@david-freistrom With Notation v1.3.1, please try adding flag --force-referrers-tag=false in your notation sign command. With this flag, you won't get the second artifact after signing. (you may also want to try Notation v2.0.0-alpha.1, which doesn't require this flag to do so anymore). But as I mentioned, this probably won't fix the no signature error during verification, because of GCP side issue.

[nobbs]

I can add some more context here, all of the commands run with notation v2.0.0-alpha.1 (redacted possibly sensitive information):

$ docker push europe-west3-docker.pkg.dev/<redacted>/<redacted>/notation-test:signed
The push refers to repository [europe-west3-docker.pkg.dev/<redacted>/<redacted>/notation-test]
...
signed: digest: sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387 size: 1990

$ notation sign <redacted> europe-west3-docker.pkg.dev/<redacted>/<redacted>/notation-test:signed@sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387
Successfully signed europe-west3-docker.pkg.dev/<redacted>/<redacted>/notation-test@sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387
Pushed the signature to europe-west3-docker.pkg.dev/<redacted>/<redacted>/notation-test@sha256:ec8a92443190e3921bea2b286fa78e67d68118b927cb0d38f4a70cc6c855ccb4

$ notation inspect europe-west3-docker.pkg.dev/<redacted>/<redacted>/notation-test@sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387
europe-west3-docker.pkg.dev/<redacted>/<redacted>/notation-test@sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387 has no associated signature

# use -d to get the requests from the inspect command, manually calling the API for referrers:
$ notation inspect europe-west3-docker.pkg.dev/<redacted>/<redacted>/notation-test@sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387 -d
...
INFO[2025-03-14T15:54:04+01:00] Reference sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387 resolved to manifest descriptor: {MediaType:application/vnd.docker.distribution.manifest.v2+json Digest:sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387 Size:1990 URLs:[] Annotations:map[] Data:[] Platform:<nil> ArtifactType:}
DEBU[2025-03-14T15:54:04+01:00] Request #3
> Request: "GET" "https://europe-west3-docker.pkg.dev/v2/<redacted>/<redacted>/notation-test/referrers/sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387?artifactType=application%2Fvnd.cncf.notary.signature"
> Request headers:
   "User-Agent": "notation/2.0.0-alpha.1"
   "Authorization": "*****"
DEBU[2025-03-14T15:54:04+01:00] Response #3
< Response status: "200 OK"
< Response headers:
   "Content-Length": "569"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Oci-Filters-Applied": "application/vnd.cncf.notary.signature"
   "X-Frame-Options": "SAMEORIGIN"
   "X-Xss-Protection": "0"
   "Alt-Svc": "h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000"
   "Content-Type": "application/vnd.oci.image.index.v1+json"
   "X-Content-Type-Options": "nosniff"
   "Date": "Fri, 14 Mar 2025 14:54:04 GMT"
europe-west3-docker.pkg.dev/<redacted>/<redacted>/notation-test@sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387 has no associated signature

$ https -A bearer -a "$TOKEN" "https://europe-west3-docker.pkg.dev/v2/<redacted>/<redacted>/notation-test/referrers/sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387?artifactType=application%2Fvnd.cncf.notary.signature"

{
    "manifests": [
        {
            "annotations": {
                "io.cncf.notary.x509chain.thumbprint#S256": "[\"fce23a8f1866993262dc397c2aa5c476fb664418ea0f049abc6dc92235ffb952\",\"7f5c0522901d9e587443510c532424c3e96839a2be90a775d328243b7ee39226\",\"e8b5edbbae77d28ab0cb58656303e843b906c38d97c5673febf008ca83562c40\"]",
                "org.opencontainers.image.created": "2025-03-14T14:53:42Z"
            },
            "digest": "sha256:ec8a92443190e3921bea2b286fa78e67d68118b927cb0d38f4a70cc6c855ccb4",
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 3372
        }
    ],
    "mediaType": "application/vnd.oci.image.index.v1+json",
    "schemaVersion": 2
}

Maybe this helps - to me it does look like GCP Artifact Registry in fact does support the referrers API - but I don't really have much knowledge of the OCI spec.

[FeynmanZhou]

Hi @nobbs , thanks for checking. From the signature tag name, it is a Referrers Tag Schema format. So I think GAR does not support Referrers API because notation sign fallbacked to Referrers Tag Schema. Were you managed to verify the signed image?

[ghost]

I can add some more context here, all of the commands run with notation v2.0.0-alpha.1 (redacted possibly sensitive information):

$ docker push europe-west3-docker.pkg.dev///notation-test:signed The push refers to repository [europe-west3-docker.pkg.dev///notation-test] ... signed: digest: sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387 size: 1990

$ notation sign europe-west3-docker.pkg.dev///notation-test:signed@sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387 Successfully signed europe-west3-docker.pkg.dev///notation-test@sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387 Pushed the signature to europe-west3-docker.pkg.dev///notation-test@sha256:ec8a92443190e3921bea2b286fa78e67d68118b927cb0d38f4a70cc6c855ccb4

$ notation inspect europe-west3-docker.pkg.dev///notation-test@sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387 europe-west3-docker.pkg.dev///notation-test@sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387 has no associated signature

use -d to get the requests from the inspect command, manually calling the API for referrers:

$ notation inspect europe-west3-docker.pkg.dev///notation-test@sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387 -d ... INFO[2025-03-14T15:54:04+01:00] Reference sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387 resolved to manifest descriptor: {MediaType:application/vnd.docker.distribution.manifest.v2+json Digest:sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387 Size:1990 URLs:[] Annotations:map[] Data:[] Platform: ArtifactType:} DEBU[2025-03-14T15:54:04+01:00] Request #3

Request: "GET" "https://europe-west3-docker.pkg.dev/v2///notation-test/referrers/sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387?artifactType=application%2Fvnd.cncf.notary.signature" Request headers: "User-Agent": "notation/2.0.0-alpha.1" "Authorization": "*****" DEBU[2025-03-14T15:54:04+01:00] Response #3 < Response status: "200 OK" < Response headers: "Content-Length": "569" "Docker-Distribution-Api-Version": "registry/2.0" "Oci-Filters-Applied": "application/vnd.cncf.notary.signature" "X-Frame-Options": "SAMEORIGIN" "X-Xss-Protection": "0" "Alt-Svc": "h3=":443"; ma=2592000,h3-29=":443"; ma=2592000" "Content-Type": "application/vnd.oci.image.index.v1+json" "X-Content-Type-Options": "nosniff" "Date": "Fri, 14 Mar 2025 14:54:04 GMT" europe-west3-docker.pkg.dev///notation-test@sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387 has no associated signature

$ https -A bearer -a "$TOKEN" "https://europe-west3-docker.pkg.dev/v2///notation-test/referrers/sha256:43d6319818d5abd2a1b4e6201f04a7d46f94cf24439327215ad4c12686337387?artifactType=application%2Fvnd.cncf.notary.signature"

{ "manifests": [ { "annotations": { "io.cncf.notary.x509chain.thumbprint#S256": "["fce23a8f1866993262dc397c2aa5c476fb664418ea0f049abc6dc92235ffb952","7f5c0522901d9e587443510c532424c3e96839a2be90a775d328243b7ee39226","e8b5edbbae77d28ab0cb58656303e843b906c38d97c5673febf008ca83562c40"]", "org.opencontainers.image.created": "2025-03-14T14:53:42Z" }, "digest": "sha256:ec8a92443190e3921bea2b286fa78e67d68118b927cb0d38f4a70cc6c855ccb4", "mediaType": "application/vnd.oci.image.manifest.v1+json", "size": 3372 } ], "mediaType": "application/vnd.oci.image.index.v1+json", "schemaVersion": 2 } Maybe this helps - to me it does look like GCP Artifact Registry in fact does support the referrers API - but I don't really have much knowledge of the OCI spec.

@nobbs The screenshot you shared shows clear referrers tag schema (OCI 1.0). For referrers api (OCI 1.1), you won't get the extra image index with tag sha256-43d... after signing. The following inspect command returns no signature associated, which is the same issue that's David is having. Under Notation 2.0.0-alpha.1, with the sign command you provided above, Notation would directly use referrers api (OCI 1.1) as long as the registry itself is OCI 1.1 compliant. Both Notation 1.3.1 and 2.0.0-alpha.1 are tested on other OCI 1.1 compliant registries without this issue.

So based on information gathered in this issue, I have the following thoughts:

1. GCP does NOT support referrers api. The reason is that during notation sign with Notation 2.0.0-alpha.1, referrers tag schema image index is still generated as shown in your screenshot. This won't happen in other OCI 1.1 compliant registries.
2. Although they claimed to be, our issue suggests GCP is actually not OCI 1.1 compliant. Because during notation verify/list/inspect, the referrers api fall back to referrers tag schema mechanism (OCI 1.1) didn't work and hence the no signature associated issue. Again, this won't happen in other OCI 1.1 compliant registries.
3. To use Notation + GCP, you may try Notation v1.1.2 (OCI 1.0).

[FeynmanZhou]

Hi @nobbs , just wanted to follow up, has this issue been resolved?

[nobbs]

Hey @FeynmanZhou - unfortunately, not in a way that allows us to continue with notation.

While we could use an older release of notation to sign images (v1.1.2 does work), we can't verify those signatures as part of pod admission. We use Kyverno as our policy engine of choice. While Kyverno does support notation natively, they mostly fall back to the implementation provided in notation-go and thus verification also fails there. Maybe running an older version of Kyverno would fix that problem, but that's not really something we want or can do.

For the time being we've decided to use cosign.

[FeynmanZhou]

Thanks @nobbs for share these information. We will test the Kyverno v1.13.0/v2.0.0-alpha.1 with Notation v1.3.1 and make sure they can work with each other. Linking https://github.com/kyverno/website/issues/1498

[baarde]

I've looked into this issue and here are my findings.

Google Artifact Registry does in fact implement the Referrers API

<Edit> …but it doesn't say so.

Notation does indeed fall back to the referrers tag schema (a previous version of this comment stated otherwise, my mistake). The reason is not the absence of the Referrers API but the fact that Google Artifact Registry forgets to report that is proceeded the subject field by including the OCI-Subject header in its response.

I have reported this as a bug on Google's issue tracker: https://issuetracker.google.com/issues/409069730 </Edit>

The Referrers API is present and does return an image index with all the referrers for the requested image that match the specified artifact type.

There's just one small problem.

⬇⬇⬇

Google Artifact implementation of Referrers API is buggy

1. Google Artifact Registry doesn't fall back to config.mediaType for artifactType

Notation signature specification was defined before the final revision of OCI 1.1. Therefore, the signature manifest doesn't include the artifactType field (#975). This shouldn't be an issue as the OCI 1.1 spec requires the server to fall back to config.mediaType when artifactType is absent from the manifest. But Google Artifact Registry doesn't do that.

So there is no artifactType field is the Referrers API response and the referrer isn't recognized as a Notation signature.

I have reported this as a bug on Google's issue tracker: https://issuetracker.google.com/issues/408967798

2. Google Artifact Registry returns an incorrect value for header OCI-Filters-Applied

On its own, the previous issue shouldn't prevent the signature to be found because the referrers are filtered. When listing signatures, the artifactType query parameter is set to application/vnd.cncf.notary.signature so the response is expected to only include Notation signatures, meaning the missing artifactType field can be ignored. However, the client can only do that when it's sure that the filtering happened on server, which means the HTTP response needs to include the OCI-Filters-Applied: artifactType header. And Google Artifact Registry doesn't do that either.

The HTTP response contains the OCI-Filters-Applied: application/vnd.cncf.notary.signature header, which is incorrect.

I have reported this as a bug on Google's issue tracker: https://issuetracker.google.com/issues/408843435

3. Google Artifact Registry returns an incorrect manifest size

So we have two bugs which combined together break signatures that rely on the config.mediaType fallback mechanism. While we're waiting for Google to fix either or both of those bugs, let's add an artifactType to the signature manifest and see what happens.

notation ls europe-docker.pkg.dev/<redacted>/alpine@sha256:1c4eef651f65e2f7daee7ee785882ac164b02b78fb74503052a26dc061c90474
europe-docker.pkg.dev/<redacted>/alpine@sha256:1c4eef651f65e2f7daee7ee785882ac164b02b78fb74503052a26dc061c90474
└── application/vnd.cncf.notary.signature
    └── sha256:e2a3a409cb425c54832de2b93af9947aa0c9c367622c27c017fc4aff9a0b39f0

Hurray, it works!

Except that…

Bad things come in threes.

notation inspect europe-docker.pkg.dev/<redacted>/alpine@sha256:1c4eef651f65e2f7daee7ee785882ac164b02b78fb74503052a26dc061c90474
Warning: unable to fetch signature sha256:e2a3a409cb425c54832de2b93af9947aa0c9c367622c27c017fc4aff9a0b39f0 due to error: GET "https://europe-docker.pkg.dev/v2/<redacted>/alpine/manifests/sha256:e2a3a409cb425c54832de2b93af9947aa0c9c367622c27c017fc4aff9a0b39f0": mismatch Content-Length
europe-docker.pkg.dev/<redacted>/alpine@sha256:1c4eef651f65e2f7daee7ee785882ac164b02b78fb74503052a26dc061c90474 has no associated signature
Error: at least one signature was skipped and not displayed

The response from the Referrers API contains a descriptor for the signature manifest. This descriptor must include the size of the manifest. And the value returned by Google Artifact Registry is, you guessed it, incorrect, which causes a mismatch Content-Length error when inspecting the signature (oras-project/oras#1626).

Here is the bug report on Google's issue tracker: https://issuetracker.google.com/issues/398888523

Conclusion

I had high hopes that implementing #975 would solve the problem without having to wait for Google to fix their broken implementation of Referrers API. So I'm a little bit disappointed. But I least now I know about OCI that I ever thought I would.

[FeynmanZhou]

@baarde Thanks for such detailed test and summary. It's really helpful! Now identified that this issue was caused by GAR's OCI spec 1.1's implementation.

[baarde]

I've corrected a small mistake in my original comment. I had missed the fact that the fallback to the referrers tag schema is handled automatically by ORAS manifest upload (as required by the spec) and happens because of yet another bug in Google Artifact Registry implementation. 🤦‍♂️

[github-actions[bot]]

This issue is stale because it has been opened for 60 days with no activity. Remove stale label or comment. Otherwise, it will be closed in 30 days.

[FeynmanZhou]

Hi @baarde , is this issue still existed? Did you receive any updates from GAR?

[baarde]

No, I haven't received any updates. Google hasn't fixed or even acknowledged any of the issues.

[tom-dudley]

I followed up the Content-Length issue with Google through other channels and believe this is now fixed (oras cp -r which previously failed, now works). I'm unsure if this unblocks this issue in any regard, but FYI.

[tom-dudley]

I followed up with Google on the OCI-Subject header and can confirm this is now fixed, so oras cp -r <src> <dst> behaves correctly on Artifact Registry now (i.e. it doesn't create the fallback tagged artifact when it shouldn't). Again, unsure how much this helps here, but FYI.

[github-actions[bot]]

This issue is stale because it has been opened for 60 days with no activity. Remove stale label or comment. Otherwise, it will be closed in 30 days.

[github-actions[bot]]

Issue closed due to no activity in the past 30 days.