pull an image from private ECR registry - containerd example?

[jkwasniak]

Summary

I looked at the introduction (https://microk8s.io/docs/registry-private "Secure Registry"), but I don't get a pull from AWS ECR.

I am currently using MicroK8s version 1.27. ECR private registry: https://XXXXXXXXXXXX.dkr.ecr.eu-central-1.amazonaws.com

I have configured /var/snap/microk8s/current/args/containerd-template.toml mirror like

[plugins.cri.registry.mirrors.”XXXXXXXXXXXX.dkr.ecr.eu-central-1.amazonaws.com"]
     endpoint = [“https://XXXXXXXXXXXX.dkr.ecr.eu-central-1.amazonaws.com"]

and auth

# Configure Registry Credentials
[plugin."io.containerd.grpc.v1.cri".registry.configs."XXXXXXXXXXXX.dkr.ecr.eu-central-1.amazonaws.com".auth]
  username = ""
  password = ""
  auth = ""
  identitytoken = ""

The question I ask myself, what must be in username, password, auth, identitytoken?

aws_access_key_id = AAAAAAAAAAAAAAAAAA aws_secret_access_key = XXXXXXXXXXXXXX

Is there an example of how to configure this right now for AWS ECR?

Thanks

[neoaggelos]

Hi @jkwasniak

Sorry for missing this. Looking at https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html, I imagine you should use your access key and secret key to retrieve an authentication token, and specify that token to the containerd configuration?

I have not tested this, though, so please let us know if it works. Thanks!

[rubel-ahammad]

Yes this worked for me. Please follow this link: https://github.com/canonical/microk8s/issues/4371#issuecomment-1917579814

On Tue, 6 Feb 2024, 5:32 pm sysarch-repo, @.***> wrote:

Hi @jkwasniak https://github.com/jkwasniak, @neoaggelos https://github.com/neoaggelos, @rubel-ahammad https://github.com/rubel-ahammad, any update on this topic?

I have successfully managed to install the ecr-credential-provider 1.28 and the associated credential-provider-config.yaml, I have also extended the kubelet config (/var/snap/microk8s/current/args/kubelet ) with the location of both files.

By setting the AWS_DEFAULT_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY I have successfully validated the setup using the CLI:

echo '{ "apiVersion": "credentialprovider.kubelet.k8s.io/v1", "kind": "CredentialProviderRequest", "image": "<ID>.dkr.ecr.<REGION>.amazonaws.com/<REPO>/<IMAGE>:<VERSION>" }' | /var/snap/microk8s/common/bin/ecr-credential-provider

and successfully obtained the login:

2024/02/06 10:30:21.339853 system_key.go:253: running from non-installed location /var/snap/microk8s/common/bin/ecr-credential-provider: ignoring system-key I0206 10:30:21.387820 11609 main.go:126] Getting creds for private registry <ID>.dkr.ecr.<REGION>.amazonaws.com/<REPO>/<IMAGE>:<VERSION> {"kind":"CredentialProviderResponse","apiVersion":"credentialprovider.kubelet.k8s.io/v1","cacheKeyType":"Registry","cacheDuration":"6h0m0s","auth":{"<ID>.dkr.ecr.<REGION>.amazonaws.com":{"username":"AWS","password":"..."}}}

But while this works using CLI, the container image cannot be pulled by microk8s, i.e. by the running kubelet / kubelite. The response code is 401 unauthorized and from the logs I see that the kubelite process does not have access to the credentials (as it is the case on CLI when the env variables are not set).

Feb 6 10:31:23 ip-10-0-0-69 microk8s.daemon-kubelite[4771]: E0206 10:31:23.079691 13236 main.go:261] Error running credential provider plugin: NoCredentialProviders: no valid providers in chain. Deprecated.

Do you know how to configure microk8s with the env variables or pointing it to ~/.aws or configuring something else so the kubelet/kubelite process can access AWS ECR and pull the images of containers running on the microk8s node?

I see a successful case documented here: #4371 https://github.com/canonical/microk8s/issues/4371 but unfortunately the aspects about the microk8s configuration are missing in that ticket.

— Reply to this email directly, view it on GitHub https://github.com/canonical/microk8s/issues/4059#issuecomment-1929328494, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADZZLWXDOR6HADWBY2UQVCDYSIID7AVCNFSM6AAAAAAZ3JUDO6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRZGMZDQNBZGQ . You are receiving this because you were mentioned.Message ID: @.***>

[sysarch-repo]

@rubel-ahammad, thanks for pointing to the ticket showing the solution using the ecr-credential-provider. That approach has worked also for me - with the following detail: the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY had to be added in the credential-provider-config.yaml env section as follows:

            apiVersion: kubelet.config.k8s.io/v1
            kind: CredentialProviderConfig
            providers:
              - name: ecr-credential-provider
                apiVersion: credentialprovider.kubelet.k8s.io/v1
                matchImages:
                  - "*.dkr.ecr.*.amazonaws.com"
                defaultCacheDuration: 12h
                env:
                  - name: AWS_SECRET_ACCESS_KEY
                    value: ...
                  - name: AWS_ACCESS_KEY_ID
                    value: ...