[ECR]: Pull through cache for authenticated registries

[srrengar]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request ECR launched pull through cache recently which reliably and anonymously retrieves public images from ECR Public and Quay.io, caches them in a customer's private ECR registry, and keeps the images in sync. Customers also want the same functionality for other private registries that require authentication, and registries need accounts to have higher pull limits https://aws.amazon.com/blogs/aws/announcing-pull-through-cache-repositories-for-amazon-elastic-container-registry/

Which service(s) is this request for? ECR

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? To cache images from private registries or images from public registries that require authentication for higher pull limits. Customers need to pass credentials through ECR pull through cache for ECR to access the images from private registries or to let ECR have access to higher pull limits.

Are you currently working around this issue? Customers need to build tooling to log in to private registries, copy those images into ECR, and keep those images in sync with the upstream private registry, and maintain this entire system.

[benjimin]

Would this also enable anonymous access, proxied through the customer's VPC, and some configuration for how frequently image tags (such as latest) are polled for updates?

Maintaining our own ECR sync lambda system (driven supply-side rather than demand-side) has a tendency to counter-productively retrieve even more image versions than we actually need (e.g. skipped increments, alternate architectures).

[afirth]

Working around - dockerhub

1. adding image-pull-secret to every namespace and every service account as required. Usually after they break once, e.g. during a cluster upgrade when we have to redeploy all the managed nodes at once. Outages caused in the past by spot-termination-handler, aws-alb-ingress-controller, kube2iam (that was bad because downstream services didn't crashloop), and probably others. This exists, but ends up either patching all serviceAccounts including the AWS and K8s ones like kube-system/statefulset and kubesystem/aws or needing manual config for every service anyway.
2. Moving images to ECR. This is mostly ok except that
   1. we build with skaffold, and the ECR login system doesn't work with the docker-api, so need to override that to use docker-cli
   2. devs adding services from helm/upstream are typically unaware of the imagePullSecret requirement (or alternatively image registry, if separately exposed in the chart) << biggest problem
   3. dependabot setup is not so straightforward

2.ii. would be solved if EKS/ECR supported setting something like

{
  "registry-mirrors": ["https://mirror.gcr.io"]
}

ideally 1) without editing bootstrap data (just slap it in here :trollface:) and 2) without using gcr

maybe it does and I just haven't found the docs.

[joebowbeer]

Being able to configure our ECR to pull-through our private GitHub Container Registry (ghcr) would be super.

[mwos-sl]

One more use case ew have is several deployments across different aws regions. We need a pull through cache in each each region for images produced by CI into one of the regions, for services like ECS and EKS. Why not ECR replication? Costs. We don't need all of the images in each region to be replicated, only used one.

[wosiu]

Excited to see "Work in Progress" label! @coultn is there any ETA for this to be GA? Asking because we've got very hot discussion about alternatives in our company. ETA could help to judge whether we should wait for a native aws solution or invest in developing some workaround in-house.