List command relies on a region being specified

[tom-pryor]

Output of docker-credential-ecr-login list gives:

Could not list credentials: MissingRegion: could not find region configuration:

Using IAM role to authenticate. Would expect the list command to not require a region as I don't see how you can set this via docker config option.

[tom-pryor]

Just as some background, we're trying to use the credential helper on our Gitlab CI runners so there is no interactive session to specify the region. You can see how they have implemented it here:

https://gitlab.com/Fodoj/gitlab-ci-multi-runner/blob/d2191e85e4804d2bc952dedd03d772d76439908e/helpers/docker/auth_config.go

Appears to be using the docker SDK to list all credentials that the helpers can provide.

From #28 it appears you have to set the region via an environment variable - but I don't see how this is possible when the credential helper is executed in this manner.

Also as a side note I've tried specifying the region in ~/.aws/config but this doesn't work either - it only gets detected if I manually execute the credential helper in a shell.

[HrmesWorld]

I also have this problem. some machine is ok, other see "Could not list credentials: MissingRegion: could not find region configuration". The config is the same.

[HrmesWorld]

Hi, I find the way to solve this issue.

1. you finish install the awscli docker-credential-ecr-login.Use awscli configure , will create config credentials at ~/.aws.

2. use aws ecr get-login --no-include-email --region cn-north-1 docker login -u AWS -p xxxxxxx https://xxxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn it will create the file ~/.docker/config.json

3. cat /etc/null > ~/.docker/config/json modify the ~/.docker/config.json to
   { "credsStore": "ecr-login" }

4. this step is important, you must docker pull one time. docker pull xxxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/xxxxxx it will create the file ~/.ecr/cache.json

5. use the cmd docker-credential-ecr-login list {"https://xxxx.dkr.ecr.cn-north-1.amazonaws.com.cn":"AWS"}

Then you can login the ECR long long.

[gengmao]

@Tomdarkness the problem is list command doesn't provide a region info but AWS sdk requires. Without specified region, AWS sdk reads it from either AWS_REGION or ~/.aws/config if AWS_SDK_LOAD_CONFIG is set. Either way requires an env variable. You can set it in the /env/environment on Ubuntu. Get command parses the region from registry domain so doesn't require env variable. And once creds are cached, list command can use the cache, as @HrmesWorld mentioned

[tom-pryor]

@gengmao Thanks, but it seems that I can't set the environment in which the credential helper is executed. As mentioned above, the credential helper is being executed using the docker SDK via the Gitlab CI runner. I've tried setting the environmental variables in the environment in which the runner itself executes but these don't seem to get inherited by the environment the credential helper is executed and it doesn't appear that Docker exposes any way for configuring the environment.

Ideally it would be great if the list command could just automatically return all registries, regardless of region, the current credentials can access.

[tom-pryor]

Or at least provide a way of configuring the region without having to manipulate the environment the helper is executed in.

[samuelkarp]

@Tomdarkness Apologies for the delayed response here. As @gengmao explained, the credential helper needs to know what region it should use in order to get credentials for the default registry in that region. However, if you know the set of registries you'll be authenticating against, you can work around this issue and configure the credential helper to use those registries in the ~/.ecr/cache.json file.

Ideally it would be great if the list command could just automatically return all registries, regardless of region, the current credentials can access.

This isn't possible. The set of registries that a given set of credentials can access is dependent both on policies applied to the credentials as well as policies applied to the resources that you're accessing (the repository in question); the policies can also include conditionals that allow access under different scenarios (time-based access, requiring MFA, etc). The IAM policy reference has a fairly detailed explanation of different ways you can configure policies. Additionally, if ECR did have a way to enumerate the access that a set of credentials has, it would risk exposing customers who had accidentally configured their repositories to allow more access than they had desired.

With that said, I do think there are a few ways we can have better default behavior:

1. Read the region from ~/.aws/config regardless of whether AWS_SDK_LOAD_CONFIG is set. I think this is a safe, non-breaking change because the only time we would use that region is when region is not already configured a different way (through AWS_REGION or through inspecting the registry URI).
2. Enumerate the known regions from the SDK and use all of them. This will enable the credential helper to know about registries in other regions than the default one in ~/.aws/config, but has two downsides: (1) it would take longer to get credentials for all regions, especially if you're not using a bunch of them, and (2) the list in the SDK can get outdated and would require updating the SDK in order to get an updated list.

I believe we'd be open to a pull request implementing the first approach above and would be happy to have discussion about whether the second approach is appropriate or not.

[psyvision]

@Tomdarkness As I said on GitLab Runner - I've got this working.

I don't know if you want to continue working towards getting it working with extra config before modifying this helper?

[joshk0]

I don't really understand the logic from @samuelkarp here. To me, the critical step is step 4 from @HrmesWorld (i actually don't need to run aws ecr get-login as it says in his post):

this step is important, you must docker pull one time. docker pull xxxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/xxxxxx it will create the file ~/.ecr/cache.json

My experience of this bug is I can't build a Dockerfile which uses FROM xxxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/xxxxxx without first running docker pull xxxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/xxxxxx on the command line. Both are running as the same user. why can't the credential helper use the same flow as it does in docker pull when it runs docker build?

[mims92]

@joshk0 Did you solve your issue? I've the same problem (using GitLab CI).

I run GitLab CI script which uses Docker. Inside the docker container, I'm trying to build an image but it fails at FROM xxx.dkr.ecr.eu-central-1.amazonaws.com.

[yueli9]

I met the same problem as @joshk0 , I have to set AWS_REGION=us-west-2 in order to fix this.

[luizhpriotto]

I had the same issue. I did this and work:

#$(aws ecr get-login --no-include-email --region sa-east-1)

#docker pull xxxxx.dkr.ecr.sa-east-1.amazonaws.com:xxxxxx

modify the ~/.docker/config.json to { "credsStore": "ecr-login" }

and again:

#docker pull xxxxx.dkr.ecr.sa-east-1.amazonaws.com:xxxxxx

#docker-credential-ecr-login list

you should get:

{"https://xxxxxx.dkr.ecr.sa-east-1.amazonaws.com":"AWS"}

it will create the file ~/.ecr/cache.json

thanks @HrmesWorld !

[deweistarsky]

Running docker pull first before docker build worked for me. Otherwise it would show a no auth error.

[jim80net]

For y'all gitlab users with ecr-login failing unless the token is cached, check out this issue to see if it's related. I just submitted a merge request that I think may help.

[CarlosDomingues]

I've been bitten by this when trying to setup Gitlab Runner using an official repository.

The installation process, by default, executes gitlab-runner as a service calling the binary directly:

[ssm-user@<ip> bin]$ sudo systemctl cat gitlab-runner | grep ExecStart
ExecStart=/usr/lib/gitlab-runner/gitlab-runner "run" "--working-directory" "/home/gitlab-runner" "--config" "/etc/gitlab-runner/config.toml" "--service" "gitlab-runner" "--syslog" "--user" "gitlab-runner"

Hence, providing environment variables via /etc/environment, /etc/profile, etc... won't work. Instead, I added AWS_REGION to my unit file directly:

sudo mkdir --parents /etc/systemd/system/gitlab-runner.service.d/
sudo touch /etc/systemd/system/gitlab-runner.service.d/local.conf
sudo tee /etc/systemd/system/gitlab-runner.service.d/local.conf > /dev/null << EOL
[Service]
Environment="AWS_REGION=$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document|jq -r .region)"
EOL
sudo systemctl daemon-reload
sudo systemctl restart gitlab-runner

Which made everything work without having to executedocker pull or docker login beforehand.

[man-jiteshm-sportsbet]

can i suggest that the plugin should also honor if AWS_DEFAULT_REGION variable is present rather than only searching for AWS_REGION.

[samuelkarp]

can i suggest that the plugin should also honor if AWS_DEFAULT_REGION variable is present rather than only searching for AWS_REGION.

@man-jiteshm-sportsbet I'd be happy to take a pull request to that effect.

[pbzdyl]

I have the same issue. I would think that if docker pull works without setting AWS_REGION env var, docker build with FROM referencing the same image as the docker pull command would work fine.

$ head -n 1 Dockerfile
FROM XXXXXXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/yyyyyyyyyyyyyyy

$ docker build .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM XXXXXXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/yyyyyyyyyyyyyyy
Get https://XXXXXXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/v2/yyyyyyyyyyyyyyy/manifests/rev-938a6f692e2345783e1f85b779533b8d792c704f: no basic auth credentials

$ cat ~/.ecr/log/ecr-login.log
time="2020-06-08T11:32:43Z" level=debug msg="Listing credentials"
time="2020-06-08T11:32:43Z" level=debug msg="No credential cache"
time="2020-06-08T11:32:43Z" level=debug msg="Calling ECR.GetAuthorizationToken for default registry"
time="2020-06-08T11:32:43Z" level=debug msg="Couldn't get authorization token" error="ecr: Failed to get authorization token: MissingRegion: could not find region configuration"

$ rm ~/.ecr/log/ecr-login.log

$ AWS_REGION=eu-west-1 docker build .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM XXXXXXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/yyyyyyyyyyyyyyy
rev-938a6f692e2345783e1f85b779533b8d792c704f: Pulling from yyyyyyyyyyyyyyy
df20fa9351a1: Pull complete
9ac7270d90d5: Pull complete
24a6f0e24dab: Pull complete
1cb78b5b90b9: Pull complete
ce7329cc5c37: Pull complete
d6fe8c1a16f0: Pull complete
Digest: sha256:21d453d7f823a9a5d33faca30059ebce025b4c871b9cca1e98091b6975e0adc6
Status: Downloaded newer image for XXXXXXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/yyyyyyyyyyyyyyy

$ cat ~/.ecr/log/ecr-login.log
time="2020-10-02T07:43:11Z" level=debug msg="Listing credentials"
time="2020-10-02T07:43:11Z" level=debug msg="No credential cache"
time="2020-10-02T07:43:11Z" level=debug msg="Calling ECR.GetAuthorizationToken for default registry"
time="2020-10-02T07:43:11Z" level=debug msg="Saving credentials to file cache" registry=XXXXXXXXXXXX
time="2020-10-02T07:43:11Z" level=debug msg="Retrieving credentials" region=eu-west-1 registry=XXXXXXXXXXXX serverURL="https://XXXXXXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com"
time="2020-10-02T07:43:11Z" level=debug msg="Checking file cache" registry=XXXXXXXXXXXX
time="2020-10-02T07:43:11Z" level=debug msg="Using cached token" registry=XXXXXXXXXXXX

[samuelkarp]

I've opened https://github.com/awslabs/amazon-ecr-credential-helper/issues/251 to track the need for better documentation of limitations related to docker build.

[rajaie-sg]

Was running into this issue, but setting the registries explicitly in the Docker config.json file fixed the issue. (From this comment)