Mirroring a repository to private ECR results in image pull failure (401 unauthorized)

[plaisted]

Image I'm using: eks 1.23 bottlerocket 1.9.2-b8074d44

What I expected to happen: Adding a repository mirror pointing to private ECR registry should successfully pull images.

What actually happened: Images fail to pull with 401 unauthorized.

How to reproduce the problem:

1. Add registry mirror to user data for bottlerocket instance template pointing an external registry to the appropriate ECR endpoint.

[[settings.container-registry.mirrors]]
  registry = "example.external.com"
  endpoint = [ "https://AWSACCOUNTID.dkr.ecr.us-east-1.amazonaws.com/example.external.com" ]

2. Attempt to run an image from using ECR image referenced directly eg kubectl -n default run -it --image AWSACCOUNTID.dkr.ecr.us-east-1.amazonaws.com/example.external.com/image:TAG test-run -> SUCCESS
3. Attempt to run an image from using the external URL mirrored internally eg kubectl -n default run -it --image example.external.com/image:TAG test-run -> FAILURE, note error message is correctly using the mirrored URL, just failing authentication.

...pulling from host AWSACCOUNTID.dkr.ecr.us-east-1.amazonaws.com failed with status code [manifests TAG]: 401 Unauthorized

Is there a way to get the node to authenticate with ECR properly when using a registry mirror?

[jpculp]

Hi @plaisted, thanks for reaching out. Do you still see the same behavior if endpoint is set to:

 endpoint = [ "https://AWSACCOUNTID.dkr.ecr.us-east-1.amazonaws.com" ]

?

[plaisted]

Updated to test but still same error.

From apiclient get, rebooted node for good measure as well:

    "container-registry": {
      "mirrors": [
        {
          "endpoint": [
            "https://AWSACCOUNTID.dkr.ecr.us-east-1.amazonaws.com"
          ],
          "registry": "registry.external.com"
        }
      ]
    },

Then kubectl -n default run -it --image registry.external.com/example:TAG test-registry and still seeing

failed to resolve reference ""registry.external.com/example:TAG": pulling from host AWSACCOUNTID.dkr.ecr.us-east-1.amazonaws.com failed with status code [manifests TAG]: 401 Unauthorized

[jpculp]

Since pulling from AWSACCOUNTID.dkr.ecr.us-east-1.amazonaws.com/example.external.com/image:TAG is working, but the mirrored registry.external.com is not, I wonder if some ECR-specific logic is being skipped when the registry is parsed. Credentials may need to be set manually as a result. We'll have to dig into this a bit more.

[rafilkmp3]

Yes I want to use my private ECR as a mirror to avoid traffic though nat gateway

I have here but don't seen to be using my private registry.

[[settings.container-registry.mirrors]]
  registry = "*"
  endpoint = [ "https://AWSACCOUNTID.dkr.ecr.us-east-2.amazonaws.com" ]

I have even used ecr mirror to have images in my private manually but it seems bottlerocket doesn't try to pull from it

ecr-mirror --registry-id AWSACCOUNTID copy "istio/proxyv2:1.16.*" "AWSACCOUNTID.dkr.ecr.us-east-2.amazonaws.com/istio/proxyv2"

What I think would be the desired behavior

[stmcginnis]

@etungsten did you have some experience with this type of configuration or some context? If so, would you be able to add any notes here?