[ECR Public] [request]: cross-repository blob mounts failing?

[tianon]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request I filed an issue on @estesp's manifest-tool at https://github.com/estesp/manifest-tool/issues/124, but in testing further, even docker manifest fails to successfully create the cross-repository manifest lists, so I believe this is something deeper.

As seen in my latest comment there, the short version is that:

docker manifest create public.ecr.aws/debian/debian:bullseye public.ecr.aws/debian/amd64:bullseye public.ecr.aws/debian/armel:bullseye public.ecr.aws/debian/armhf:bullseye public.ecr.aws/debian/arm64:bullseye public.ecr.aws/debian/i386:bullseye public.ecr.aws/debian/mips64el:bullseye public.ecr.aws/debian/ppc64el:bullseye public.ecr.aws/debian/s390x:bullseye

followed by:

docker manifest push public.ecr.aws/debian/debian:bullseye

leads to:

error mounting debian/amd64@sha256:4c0bfd2a9f4e90b5a07958f07d71b2e2ea5a965649c92bf35822aece72a29c6b to public.ecr.aws/debian/debian:bullseye

Which service(s) is this request for? ECR Public

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? At a high level, I have per-architecture images pushed to each of public.ecr.aws/debian/amd64, public.ecr.aws/debian/armel, etc, and I want to push the combined manifest lists to public.ecr.aws/debian/debian. Functionally, this means the clients creating the manifest list have to do cross-repository blob mounts, but for some reason that doesn't seem to be working properly in any of the three tools I've tested (which I use to do exactly this successfully against other registries).

Are you currently working around this issue? I was hoping docker manifest would be my workaround, but it looks like I'm going to have to go as far as using containerd to do push-by-digest of all the per-arch images instead. :disappointed:

cc @samuelkarp

[estesp]

Thanks for opening this @tianon. If that's the root cause, I'm curious if a tag-based multi-arch assembly would be a short-term workaround (maybe not helpful for your use case, but would enable existing tools in the short term)? If all the images are in the same repo (e.g. "target_name:version_tag") then all the member images could be pushed as target_name:archstring_version getting around the need for a blob mount.

[tianon]

Yeah, that's doable, but not really less work than doing the push-by-digest solution (and leaves the extra tags hanging around unless I clean them up after). :sweat_smile:

I'm sure if I take each image manifest and do a direct push-by-digest via ctr instead of a cross-repository mount it'd likely work, but I need to do more testing around that to be sure (that's probably my next project on this, although not sure how soon I'll get around to testing it :see_no_evil:).

FWIW, @samuelkarp was able to reproduce too:

$ docker manifest create public.ecr.aws/xxx/test:tianon  public.ecr.aws/debian/amd64:bullseye public.ecr.aws/debian/arm64:bullseye
Created manifest list public.ecr.aws/xxx/test:tianon
$ docker manifest push public.ecr.aws/xxx/test:tianon
error mounting debian/amd64@sha256:4c0bfd2a9f4e90b5a07958f07d71b2e2ea5a965649c92bf35822aece72a29c6b to public.ecr.aws/xxx/test:tianon

[tianon]

Got around to it sooner than I expected -- can confirm that pushing all the manifests by-digest to public.ecr.aws/debian/debian made manifest-tool work (although docker manifest still complains about not being able to mount -- I guess manifest-tool is more forgiving?).

[jonjohnsonjr]

Ideally ECR Public would support mounting, but it should at least return a 202 here instead of failing.

Alternatively, if a registry does not support cross-repository mounting or is unable to mount the requested blob, it SHOULD return a 202. This indicates that the upload session has begun and that the client MAY proceed with the upload..

If a mount fails due to invalid repository or digest arguments, the registry will fall back to the standard upload behavior and return a 202 Accepted with the upload URL in the Location header:

[cmorinupgrade]

Thanks for opening this @tianon. If that's the root cause, I'm curious if a tag-based multi-arch assembly would be a short-term workaround (maybe not helpful for your use case, but would enable existing tools in the short term)? If all the images are in the same repo (e.g. "target_name:version_tag") then all the member images could be pushed as target_name:archstring_version getting around the need for a blob mount.

@estesp is there any new ways to achieve this to this day, other than using the-based approach? Is there a way to enable blob mounting at ECR level? We are looking to push multi-arch manifest to our private ECR repositories and we are getting this exact error message. Thanks for your help 🙏