Unable to pull from ECR using IAM Role on Instance

[ip-sf]

I have been beating my head against a wall, and every thing i search on google is now a purple link.

I am attempting to use this in conjunction with Nomad, but I cannot even get this working, let alone the issues with Nomad.

I am using an IAM policy attached to the role that is assigned to the EC2 instance I am attempting use. The IAM Policy is the standard AmazonEC2ContainerServiceforEC2Role.

When manually logging in using aws ecr and docker login I can push/pull just fine, so I think I've ruled out IAM related issues.

aws ecr get-login-password --region us-west-2 | \
sudo docker login --username AWS --password-stdin \ 
<account>.dkr.ecr.us-west-2.amazonaws.com

sudo docker pull <account>.dkr.ecr.us-west-2.amazonaws.com/<image>:latest
latest: Pulling from <image>
42c077c10790: Pull complete
f63e77b7563a: Pull complete
0c31162eec9d: Pull complete
7cfd3784111c: Pull complete
791791ccdd73: Pull complete
ef21c12392c8: Pull complete
Digest: sha256:f43b...cded3
Status: Downloaded newer image for  <account>.dkr.ecr.us-west-2.amazonaws.com/<image>:latest
 <account>.dkr.ecr.<region>.amazonaws.com/<image>:latest

However, trying to do the same while utilizing docker-credential-ecr-login fails.

I have tried multiple variations of the following in config.json

{
    "credHelpers": {
        "<account>.dkr.ecr.us-west-2.amazonaws.com": "ecr-login"
    }
}

{
    "credsStore": "ecr-login"
}

{
    "credHelpers": {
        "<account>.dkr.ecr.us-west-2.amazonaws.com": "ecr-login"
    },
    "credsStore": "ecr-login"
}

I've tried each variation in the following locations, all with 644 perms

• /root/.docker/config.json
• /etc/docker/config.json
• ~/.docker/config.json

Despite every effort, I get the following:

docker-credential-ecr-login get <<< <account>.dkr.ecr.us-west-2.amazonaws.com
credentials not found in native keychain

cat ~/.ecr/log/ecr-login.log
time="2022-07-28T21:19:46Z" level=debug msg="Could not fetch credentials for cache prefix, disabling cache" error="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
time="2022-07-28T21:19:46Z" level=debug msg="Retrieving credentials" region=us-west-2 registry=<account> serverURL=<account>.dkr.ecr.us-west-2.amazonaws.com
time="2022-07-28T21:19:46Z" level=debug msg="Calling ECR.GetAuthorizationToken" registry=<account>
time="2022-07-28T21:19:46Z" level=error msg="Error retrieving credentials" error="ecr: Failed to get authorization token: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"

current env

SHELL=/bin/bash
PWD=/root/.docker
LOGNAME=root
HOME=/root
LANG=C.UTF-8
USER=root
LC_TERMINAL_VERSION=3.4.16
SHLVL=1
AWS_SDK_LOAD_CONFIG=true  # I've tried this true/false and unset
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
MAIL=/var/mail/root
_=/usr/bin/env

Contents of /root/.aws/config, the only file in .aws/

cat ~/.aws/config
[default]
region = us-west-2

I'm sure there is something really simple I am missing, but the error logging isn't very helpful in this case. What chain? Why is it using a chain if it doesn't need credentials because of the existing IAM policy?

TIA for any suggestions or troubleshooting advice!

[abronan]

@ip-sf Which version of docker-credential-ecr-login are you using?

You can check this running docker-credential-ecr-login -v.

I had the same error (although this was using an SSO config) that was due to an outdated version of the tool installed via apt on Ubuntu. Doing a go install on latest (following the documentation) solved it.

[robertkruk]

i used the following

code ~/.docker/config.json

edit config.json

{
	"auths": {
		"account1111.dkr.ecr.ap-southeast-2.amazonaws.com": {},
		"accountxxxx.dkr.ecr.ap-southeast-2.amazonaws.com": {},
		"registry.gitlab.com": {}
	},
	"credsStore": "ecr-login"
}

then

AWS_PROFILE=profile-name docker pull accountxxx.dkr.ecr.ap-southeast-2.amazonaws.com/foo/bar:latest

[ymguerra]

Hi @ip-sf , did you manage to solve this problem? I have the exact same issue

[ghost]

Hi @ip-sf , did you manage to solve this problem? I have the exact same issue

me too I'm dying inside

[ip-sf]

I did get it working, but I dont exactly remember what i had to do.

This is how it's configured in my nomad environment, and it is working.

I can check other config elements if needed, just let me know. Hopefully this helps. Sorry for the delay, i legit didnt see the notifications until today. I feel all of your pain and i want to help :)

Nomad Client Config

plugin "docker" {
  config {
    auth {
      config = "/etc/docker/config.json"
      helper = "ecr-login"
    }
  }
}

/etc/docker/config.json

{
    "credHelpers": {"<accountid>.dkr.ecr.us-west-2.amazonaws.com": "ecr-login"}
}

IAM Instance Profile applied to Nomad Client hosts

{
    "Statement": [
        {
            "Action": [
                "ecr:ListTagsForResource",
                "ecr:GetRepositoryPolicy",
                "ecr:GetLifecyclePolicyPreview",
                "ecr:GetLifecyclePolicy",
                "ecr:GetDownloadUrlForLayer",
                "ecr:DescribeRepositories",
                "ecr:DescribeImages",
                "ecr:DescribeImageScanFindings",
                "ecr:DescribeImageReplicationStatus",
                "ecr:BatchGetRepositoryScanningConfiguration",
                "ecr:BatchGetImage",
                "ecr:BatchCheckLayerAvailability"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:ecr:*:<accountid>:repository/*",
            "Sid": ""
        },
        {
            "Action": [
                "ecr:GetRegistryScanningConfiguration",
                "ecr:GetRegistryPolicy",
                "ecr:GetAuthorizationToken",
                "ecr:DescribeRegistry",
                "ecr:DescribePullThroughCacheRules"
            ],
            "Effect": "Allow",
            "Resource": "*",
            "Sid": ""
        }
    ],
    "Version": "2012-10-17"
}

Required config in Nomad Job

https://www.nomadproject.io/docs/drivers/docker#auth_soft_fail

job "job" {
group "group" {
    task "task" {
      driver = "docker"
      config {
        auth_soft_fail = true
      }
    }
  }
}

Script logic being used to install/configure the ECR helper (truncated)

export GO_VER="1.18.4"
export ECR_HELPER_REPO="github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login@latest"
export DEAFULT_ECR_HELPER_CFG="/etc/docker/config.json"
# Install docker-credential-ecr-login
# install go
wget -q https://go.dev/dl/go${GO_VER}.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go${GO_VER}.linux-amd64.tar.gz
sudo ln -s /usr/local/go/bin/go /usr/bin/go
go install $ECR_HELPER_REPO
# This will install to ~/go/bin, move it to somewhere $PATH will find
sudo mv go/bin/docker-credential-ecr-login /usr/bin/
# Default config.json for provisioning
sudo mkdir -p /etc/docker
sudo sh -c 'cat <<EOF >> ${DEAFULT_ECR_HELPER_CFG}
{
    "credHelpers": {"<accountID>.dkr.ecr.<region>.amazonaws.com": "ecr-login"}
}
EOF
'

[ip-sf]

I had the same error (although this was using an SSO config) that was due to an outdated version of the tool installed via apt on Ubuntu. Doing a go install on latest (following the documentation) solved it.

I do believe this was part of it. I am currently installing GO from source and then installing the helper via go directly.

See details in above post.

[ghost]

I had the same error (although this was using an SSO config) that was due to an outdated version of the tool installed via apt on Ubuntu. Doing a go install on latest (following the documentation) solved it.

I do believe this was part of it. I am currently installing GO from source and then installing the helper via go directly.

See details in above post.

@ip-sf You sir, are my hero :) I had ended up doing my own cronjob to refresh my creds, but your way is better. Thank you

[apamildner]

@ip-sf Hero status confirmed again today by solving our issue 🥳