oras Error trying to push a manifest to azure registry with retention policy

What happened in your environment?

When trying to push a manifest with a tag to an azure registry with a retention policy, oras reports a 500: Internal Server Error. If I disable the retention policy on azure, I can push the tag successfully.

Interestingly, notation is able to push its blobs and manifests to the same registry, even when the retention policy is enabled. This suggests that there is something that can be done on the client side to make it work.

Notes:

Pushing the manifest without the tag works correctly,

cat manifest | oras manifest push -d -v my-registry.azurecr.io/my/repo -

Pushing the manifest with a tag produces the output below,

cat manifest | oras manifest push -d -v my-registry.azurecr.io/my/repo:my-tag -

Pushing the manifest without a tag, and then tagging with oras tag also produces a 500 error.

The detailed output, with -d -v flags (I have added separator lines for visual comfort):

——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:28+02:00 level=debug msg=Request #0
> Request URL: "https://my-registry.azurecr.io/v2/my/repo/manifests/my-tag"
> Request method: "HEAD"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "User-Agent": "oras/1.1.0"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Response #0
< Response Status: "401 Unauthorized"
< Response headers:
   "Www-Authenticate": "Bearer realm=\"https://my-registry.azurecr.io/oauth2/token\",service=\"my-registry.azurecr.io\",scope=\"repository:my/repo:pull\""
   "X-Content-Type-Options": "nosniff"
   "Content-Type": "application/json; charset=utf-8"
   "Connection": "keep-alive"
   "Date": "Mon, 13 May 2024 21:07:29 GMT"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains, max-age=31536000; includeSubDomains"
   "Server": "AzureContainerRegistry"
   "Access-Control-Expose-Headers": "Docker-Content-Digest, WWW-Authenticate, Link, X-Ms-Correlation-Request-Id"
   "X-Ms-Correlation-Request-Id": "46c27317-b52e-4c52-a266-455d332d6646"
   "Content-Length": "217"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Request #1
> Request URL: "https://my-registry.azurecr.io/oauth2/token"
> Request method: "POST"
> Request headers:
   "Content-Type": "application/x-www-form-urlencoded"
   "User-Agent": "oras/1.1.0"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Response #1
< Response Status: "200 OK"
< Response headers:
   "Date": "Mon, 13 May 2024 21:07:29 GMT"
   "Content-Type": "application/json; charset=utf-8"
   "Connection": "keep-alive"
   "X-Ms-Correlation-Request-Id": "a68cf443-1d5f-4e66-8f5b-ff28eb9188f2"
   "X-Ms-Ratelimit-Remaining-Calls-Per-Second": "333.316667"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains"
   "Server": "AzureContainerRegistry"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Request #2
> Request URL: "https://my-registry.azurecr.io/v2/my/repo/manifests/my-tag"
> Request method: "HEAD"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "Authorization": "*****"
   "User-Agent": "oras/1.1.0"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Response #2
< Response Status: "404 Not Found"
< Response headers:
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains, max-age=31536000; includeSubDomains"
   "X-Ms-Request-Id": "704e3e35-17f4-4667-9f23-78efa2de3a00"
   "Server": "AzureContainerRegistry"
   "Content-Type": "application/json; charset=utf-8"
   "Connection": "keep-alive"
   "Access-Control-Expose-Headers": "Docker-Content-Digest, WWW-Authenticate, Link, X-Ms-Correlation-Request-Id"
   "X-Ms-Correlation-Request-Id": "c09c0f21-233b-49ce-b8d1-885f73c533bd"
   "Date": "Mon, 13 May 2024 21:07:29 GMT"
   "Content-Length": "276"
   "X-Content-Type-Options": "nosniff"
   "X-Ms-Client-Request-Id": ""
Uploading 77f69ad47d16 application/vnd.oci.image.index.v1+json
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Request #3
> Request URL: "https://my-registry.azurecr.io/v2/my/repo/manifests/my-tag"
> Request method: "PUT"
> Request headers:
   "Content-Type": "application/vnd.oci.image.index.v1+json"
   "User-Agent": "oras/1.1.0"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Response #3
< Response Status: "401 Unauthorized"
< Response headers:
   "Content-Type": "application/json; charset=utf-8"
   "Connection": "keep-alive"
   "Access-Control-Expose-Headers": "Docker-Content-Digest, WWW-Authenticate, Link, X-Ms-Correlation-Request-Id"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Server": "AzureContainerRegistry"
   "Date": "Mon, 13 May 2024 21:07:29 GMT"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains, max-age=31536000; includeSubDomains"
   "X-Content-Type-Options": "nosniff"
   "X-Ms-Correlation-Request-Id": "21cc6879-d24f-4b6e-a7e3-e7d2bc932780"
   "Content-Length": "288"
   "Www-Authenticate": "Bearer realm=\"https://my-registry.azurecr.io/oauth2/token\",service=\"my-registry.azurecr.io\",scope=\"repository:my/repo:pull,push\""
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Request #4
> Request URL: "https://my-registry.azurecr.io/oauth2/token"
> Request method: "POST"
> Request headers:
   "Content-Type": "application/x-www-form-urlencoded"
   "User-Agent": "oras/1.1.0"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Response #4
< Response Status: "200 OK"
< Response headers:
   "X-Ms-Ratelimit-Remaining-Calls-Per-Second": "333.3"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains"
   "Server": "AzureContainerRegistry"
   "Date": "Mon, 13 May 2024 21:07:29 GMT"
   "Content-Type": "application/json; charset=utf-8"
   "Connection": "keep-alive"
   "X-Ms-Correlation-Request-Id": "19fe5024-c9be-47c0-ad6f-cb7d28982426"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:29+02:00 level=debug msg=Request #5
> Request URL: "https://my-registry.azurecr.io/v2/my/repo/manifests/my-tag"
> Request method: "PUT"
> Request headers:
   "User-Agent": "oras/1.1.0"
   "Content-Type": "application/vnd.oci.image.index.v1+json"
   "Authorization": "*****"
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
time=2024-05-13T23:07:51+02:00 level=debug msg=Response #5
< Response Status: "500 Internal Server Error"
< Response headers:
   "Content-Length": "0"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "X-Ms-Client-Request-Id": ""
   "X-Ms-Request-Id": "7f778f02-d3c0-4206-aec8-69ad501a7d4a"
   "Server": "AzureContainerRegistry"
   "Connection": "keep-alive"
   "X-Content-Type-Options": "nosniff"
   "X-Ms-Correlation-Request-Id": "07bea16f-8787-4ed8-8f16-d06aefe63ca7"
   "Date": "Mon, 13 May 2024 21:07:51 GMT"
   "Content-Type": "application/json; charset=utf-8"
   "Access-Control-Expose-Headers": "Docker-Content-Digest, WWW-Authenticate, Link, X-Ms-Correlation-Request-Id"
   "Strict-Transport-Security": "max-age=31536000; includeSubDomains, max-age=31536000; includeSubDomains"
Error: failed to tag my-tag: PUT "https://my-registry.azurecr.io/v2/my/repo/manifests/my-tag": response status code 500: Internal Server Error

What did you expect to happen?

A successful response when pushing the tag.

How can we reproduce it?

Setup an azure registry and enable a retention policy. Then run

cat manifest | oras manifest push -d -v my-registry.azurecr.io/my/repo:my-tag -

What is the version of your ORAS CLI?

Version:        1.1.0
Go version:     go1.21.0
Git commit:     7079c468a06fb5815c99395eb4aaf46dd459d3fa
Git tree state: clean

What is your OS environment?

ubuntu 22.04

Are you willing to submit PRs to fix it?

[ ] Yes, I am willing to fix it.

May 16 '24 08:05 side-chains

@side-chains How is the retention policy setup? What is the retention period?

May 17 '24 08:05 qweeah

I have added separator lines for visual comfort

@FeynmanZhou A strong plus on adding such visual comfort to the HTTP trace logs.

May 17 '24 08:05 qweeah

@side-chains How is the retention policy setup? What is the retention period?

Click on the registry resource. On the left menu, under policies, there is an item named retention (preview). There you can only enable or disable the retention policy and, if it is enabled, you can choose the retention period in days.

I have tried with registries with retention periods of 5 and 7 days. The problem is the same in both cases.

May 17 '24 14:05 side-chains

Checked registry logs and it should be a bug in ACR's retention feature. @side-chains Since you already have an Azure subscription, you can file a service ticket referencing this GitHub issue and we will follow it up in the ticket for privacy purposes.

May 20 '24 13:05 qweeah

Checked registry logs and it should be a bug in ACR's retention feature. @side-chains Since you already have an Azure subscription, you can file a service ticket referencing this GitHub issue and we will follow it up in the ticket for privacy purposes.

@qweeah Does oras use the same base code as notation? Or, does it not interact with the registry in a similar way? If the bug is on azure's side, how does it explain that notation can push and tag without getting the error?

May 21 '24 08:05 side-chains

As far as I know, notation should generate artifact without tags.

May 21 '24 08:05 qweeah

As far as I know, notation should generate artifact without tags.

@qweeah Well, it creates a signature manifest, with the subject being the image that is being signed, and one or more layers with signatures. This one is, indeed, untagged.

However, it also creates an index manifest with a list of manifests pointing to all signature manifests corresponding to that image. This one is tagged as sha256- followed by the sha256 of the image that is being signed.

May 21 '24 08:05 side-chains

Okay, in that case(referrers tag schema) ORAS and notation should use the same SDK, oras-go. Will check the implementation detail later. Back to your issue, I am pretty sure that errors are caused by ACR but cannot share server-side logs here for privacy consideration.

May 21 '24 09:05 qweeah

@side-chains I have confirmed that both Notation and ORAS push manifests in the same way. Your error is caused by a known issue in ACR retention policy feature(I cannot share server-side logs here but feel free to join our slack channel https://cloud-native.slack.com/archives/CJ1KHJM5Z to discuss). I would suggest you raise a ticket and get help from ACR supports directly.

May 28 '24 06:05 qweeah

@FeynmanZhou one more usecase IMHO remove -d and linking - #911

Jun 10 '24 19:06 sajayantony

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

Aug 10 '24 01:08 github-actions[bot]

This issue was closed because it has been stalled for 30 days with no activity.

Sep 09 '24 01:09 github-actions[bot]