oras icon indicating copy to clipboard operation
oras copied to clipboard
verified publisher icon oras-project

Resursively copying a multi-plarform image might fail when the index manifest has no referrer

Open qweeah opened this issue 9 months ago • 1 comments

What happened in your environment?

Crafted below artifacts in a registry, where A is a image index with manifests field pointing a list of manifests that contains only image manifest B, and C is a referrer that points to B as its subject.

graph TD
    A -- manifests ---> B
    C -- subject --> B

When running oras cp -r with source as A, the copy process fails

What did you expect to happen?

Should succeed to copy all the three artifacts.

How can we reproduce it?

REG=localhost:6000
REPO=test
TAG=demo

SUBJECT=$(oras push $REG/$REPO -a test=true --format=go-template={{.reference}}) # generate image manifest
oras attach --artifact-type test/demo $SUBJECT -a test=true # attach a referrer to manifest
oras manifest index create  $REG/$REPO:$TAG $SUBJECT # create index
oras cp -r $REG/$REPO:$TAG --to-oci-layout sandbox:test # copy index to an OCI layout, will fail

The logs of oras cp is

[2025-05-12T00:49:35.144358495Z][DEBUG]: --> Request #0
> Request URL: "http://localhost:6000/v2/test/manifests/demo"
> Request method: "HEAD"
> Request headers:
   "User-Agent": "oras/1.3.0-beta.3"
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"


[2025-05-12T00:49:35.14791852Z][DEBUG]: <-- Response #0
< Response Status: "401 Unauthorized"
< Response headers:
   "X-Content-Type-Options": "nosniff"
   "Date": "Mon, 12 May 2025 00:49:35 GMT"
   "Content-Length": "147"
   "Content-Type": "application/json; charset=utf-8"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Www-Authenticate": "Basic realm=\"test-basic\""
< Response body:
   No response body to print


[2025-05-12T00:49:35.148017021Z][DEBUG]: --> Request #1
> Request URL: "http://localhost:6000/v2/test/manifests/demo"
> Request method: "HEAD"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "Authorization": "*****"
   "User-Agent": "oras/1.3.0-beta.3"


[2025-05-12T00:49:35.153247158Z][DEBUG]: <-- Response #1
< Response Status: "200 OK"
< Response headers:
   "Date": "Mon, 12 May 2025 00:49:35 GMT"
   "Content-Length": "240"
   "Content-Type": "application/vnd.oci.image.index.v1+json"
   "Docker-Content-Digest": "sha256:1c955ee5c1441f7588eb123554b9135aec4f3ae9cb7f91d31a1c2290a3ba4f42"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Etag": "\"sha256:1c955ee5c1441f7588eb123554b9135aec4f3ae9cb7f91d31a1c2290a3ba4f42\""
   "X-Content-Type-Options": "nosniff"
< Response body:
   No response body to print


[2025-05-12T00:49:35.153326759Z][DEBUG]: --> Request #2
> Request URL: "http://localhost:6000/v2/test/manifests/sha256:1c955ee5c1441f7588eb123554b9135aec4f3ae9cb7f91d31a1c2290a3ba4f42"
> Request method: "GET"
> Request headers:
   "Accept": "application/vnd.oci.image.index.v1+json"
   "Authorization": "*****"
   "User-Agent": "oras/1.3.0-beta.3"


[2025-05-12T00:49:35.159797405Z][DEBUG]: <-- Response #2
< Response Status: "200 OK"
< Response headers:
   "X-Content-Type-Options": "nosniff"
   "Date": "Mon, 12 May 2025 00:49:35 GMT"
   "Content-Length": "240"
   "Content-Type": "application/vnd.oci.image.index.v1+json"
   "Docker-Content-Digest": "sha256:1c955ee5c1441f7588eb123554b9135aec4f3ae9cb7f91d31a1c2290a3ba4f42"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Etag": "\"sha256:1c955ee5c1441f7588eb123554b9135aec4f3ae9cb7f91d31a1c2290a3ba4f42\""
< Response body:
{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:fd4c407654a9191baf2c0d8c0934a0162f0865593697316e65799d6e8efca569","size":549}]}


[2025-05-12T00:49:35.160008306Z][DEBUG]: --> Request #3
> Request URL: "http://localhost:6000/v2/test/referrers/sha256:fd4c407654a9191baf2c0d8c0934a0162f0865593697316e65799d6e8efca569"
> Request method: "GET"
> Request headers:
   "Authorization": "*****"
   "User-Agent": "oras/1.3.0-beta.3"


[2025-05-12T00:49:35.160633511Z][DEBUG]: <-- Response #3
< Response Status: "404 Not Found"
< Response headers:
   "Date": "Mon, 12 May 2025 00:49:35 GMT"
   "Content-Length": "19"
   "Content-Type": "text/plain; charset=utf-8"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "X-Content-Type-Options": "nosniff"
< Response body:
404 page not found



[2025-05-12T00:49:35.160706211Z][DEBUG]: --> Request #4
> Request URL: "http://localhost:6000/v2/test/manifests/sha256-fd4c407654a9191baf2c0d8c0934a0162f0865593697316e65799d6e8efca569"
> Request method: "GET"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "Authorization": "*****"
   "User-Agent": "oras/1.3.0-beta.3"


[2025-05-12T00:49:35.165550246Z][DEBUG]: <-- Response #4
< Response Status: "200 OK"
< Response headers:
   "Date": "Mon, 12 May 2025 00:49:35 GMT"
   "Content-Length": "355"
   "Content-Type": "application/vnd.oci.image.index.v1+json"
   "Docker-Content-Digest": "sha256:0eb02a9fc7e60668c1997b114c30f0594a3fead7bab35a2e38637567456274e6"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Etag": "\"sha256:0eb02a9fc7e60668c1997b114c30f0594a3fead7bab35a2e38637567456274e6\""
   "X-Content-Type-Options": "nosniff"
< Response body:
{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:b0c3f7f0add6d02caf30286ff6bedefa5f3b504477ea2e7af9c62e6ce72f7f4d","size":686,"annotations":{"org.opencontainers.image.created":"2025-05-12T00:49:04Z","test":"true"},"artifactType":"test/demo"}]}


[2025-05-12T00:49:35.165695447Z][DEBUG]: --> Request #5
> Request URL: "http://localhost:6000/v2/test/manifests/sha256:1c955ee5c1441f7588eb123554b9135aec4f3ae9cb7f91d31a1c2290a3ba4f42"
> Request method: "HEAD"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "Authorization": "*****"
   "User-Agent": "oras/1.3.0-beta.3"


[2025-05-12T00:49:35.170313679Z][DEBUG]: <-- Response #5
< Response Status: "200 OK"
< Response headers:
   "Content-Length": "240"
   "Content-Type": "application/vnd.oci.image.index.v1+json"
   "Docker-Content-Digest": "sha256:1c955ee5c1441f7588eb123554b9135aec4f3ae9cb7f91d31a1c2290a3ba4f42"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "Etag": "\"sha256:1c955ee5c1441f7588eb123554b9135aec4f3ae9cb7f91d31a1c2290a3ba4f42\""
   "X-Content-Type-Options": "nosniff"
   "Date": "Mon, 12 May 2025 00:49:35 GMT"
< Response body:
   No response body to print


[2025-05-12T00:49:35.17040918Z][DEBUG]: --> Request #6
> Request URL: "http://localhost:6000/v2/test/manifests/sha256-1c955ee5c1441f7588eb123554b9135aec4f3ae9cb7f91d31a1c2290a3ba4f42"
> Request method: "GET"
> Request headers:
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"
   "Authorization": "*****"
   "User-Agent": "oras/1.3.0-beta.3"


[2025-05-12T00:49:35.174777111Z][DEBUG]: <-- Response #6
< Response Status: "404 Not Found"
< Response headers:
   "X-Content-Type-Options": "nosniff"
   "Date": "Mon, 12 May 2025 00:49:35 GMT"
   "Content-Length": "161"
   "Content-Type": "application/json; charset=utf-8"
   "Docker-Distribution-Api-Version": "registry/2.0"
< Response body:
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"sha256-1c955ee5c1441f7588eb123554b9135aec4f3ae9cb7f91d31a1c2290a3ba4f42"}}]}



[2025-05-12T00:49:35.174862712Z][DEBUG]: --> Request #7
> Request URL: "http://localhost:6000/v2/test/manifests/sha256-b0c3f7f0add6d02caf30286ff6bedefa5f3b504477ea2e7af9c62e6ce72f7f4d"
> Request method: "GET"
> Request headers:
   "Authorization": "*****"
   "User-Agent": "oras/1.3.0-beta.3"
   "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json"


[2025-05-12T00:49:35.179291343Z][DEBUG]: <-- Response #7
< Response Status: "404 Not Found"
< Response headers:
   "Content-Type": "application/json; charset=utf-8"
   "Docker-Distribution-Api-Version": "registry/2.0"
   "X-Content-Type-Options": "nosniff"
   "Date": "Mon, 12 May 2025 00:49:35 GMT"
   "Content-Length": "161"
< Response body:
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Tag":"sha256-b0c3f7f0add6d02caf30286ff6bedefa5f3b504477ea2e7af9c62e6ce72f7f4d"}}]}



Exists  b0c3f7f0add6 application/vnd.oci.image.manifest.v1+json
Error response from registry: failed to perform "Tag" on destination: sha256:1c955ee5c1441f7588eb123554b9135aec4f3ae9cb7f91d31a1c2290a3ba4f42: application/vnd.oci.image.index.v1+json: not found

What is the version of your ORAS CLI?

Version: 1.3.0-beta.3 Go version: go1.24.2 OS/Arch: linux/amd64 Git commit: fce0546a012909e4d7234677f92a85b482030e39 Git tree state: clean

What is your OS environment?

Ubuntu 20.04

Are you willing to submit PRs to fix it?

  • [ ] Yes, I am willing to fix it.

qweeah avatar May 12 '25 00:05 qweeah

The bug is introduced in #1122, below is an analysis of what happened based on copying A recursively:

graph TD
    A -- manifests ---> B
    C -- subject --> B

When initiating the extended copy, oras-go will launch a scan to find all root nodes. Only nodes with 0 incoming edges will be consider as root. Such scanning is based on the provided starting point, which is A.

To support copying all referrers of images in the index, oras will customize the FindPredecessors handler in the copy options to set C as a predecessor of A.

graph BT
    A -- predecessor --> C
    B -- predecessor --> C

Based on the above graph, such root scan will only consider C as a root. So the extended copy only covers accessible nodes from C, which is

graph TD
    C -- subject --> B

And A is not copied.

qweeah avatar May 12 '25 00:05 qweeah