amazon-ecs-local-container-endpoints
amazon-ecs-local-container-endpoints copied to clipboard
awslabs
NoCredentialProviders: no valid providers in chain exception
Hi there, can someone please help me understand what I'm doing wrong. I'm following the docs to make local env setup working.
Here is my override file
version: "2"
networks:
credentials_network:
driver: bridge
ipam:
config:
- subnet: "169.254.170.0/24"
gateway: 169.254.170.1
services:
ecs-local-endpoints:
image: amazon/amazon-ecs-local-container-endpoints
volumes:
- /var/run:/var/run
- "~/.aws:/home/.aws:ro"
# env_file:
# - ~/.clokta/fusion.env
environment:
HOME:
AWS_PROFILE: "fusion"
networks:
credentials_network:
ipv4_address: "169.254.170.2"
proxy:
depends_on:
- ecs-local-endpoints
networks:
credentials_network:
ipv4_address: "169.254.170.3"
environment:
AWS_DEFAULT_REGION: "us-east-1"
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/role/blah-role"
ECS_CONTAINER_METADATA_URI: "http://169.254.170.2/v3"
I've logged in through SSO and have STS issued credentials. With this set up I'm making the following request:
GET 169.254.170.2/role/blah-role HTTP/1.1
Host:169.254.170.2
Connection: close
As a result, I'm getting the following response:
HTTP/1.1 500 Internal Server Error
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 25 Feb 2020 19:06:08 GMT
Content-Length: 435
Connection: close
Internal Server Error: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, fusion.
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: connection refused
ecs local container logs has this line:
ecs-local-endpoints_1 | time="2020-02-25T19:06:07Z" level=error msg="HTTP 500 - NoCredentialProviders: no valid providers in chain\ncaused by: EnvAccessKeyNotFound: failed to find credentials in the environment.\nSharedCredsLoad: failed to load profile, fusion.\nEC2RoleRequestError: no EC2 instance role found\ncaused by: RequestError: send request failed\ncaused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: connection refused"
However, if I uncomment
# env_file:
# - ~/.clokta/fusion.env
and rerun docker-compose, everything seems fine.
~/.clokta/fusion.env contains AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. Those values match the [fusion] section of the ~/.aws/credentials which I'm mapping to the container, except aws_access_key_id key and friends are in the lower case.
Is there something I'm missing in the setup?
Hi @yury-sannikov - Are you still having this issue? I'm so far unable to reproduce this but am still working on it.
@yury-sannikov I noticed two things in your compose file:
volumes:
- /var/run:/var/run
- "~/.aws:/home/.aws:ro"
# env_file:
# - ~/.clokta/fusion.env
environment:
HOME:
AWS_PROFILE: "fusion"
- Why is the home volume mount: "/home/.aws:ro" - what's the ":ro"? That seems weird.
- Set a value for HOME:
HOME: /home
Edit: I have been told that the "ro" is for read only. Derp.
I have the same issue, my configuration is similar to what @yury-sannikov, not sure if that is the same problem, but I'm behind a corporate proxy. I setup my no_proxy to have both "169.254.170.2, 169.254.169.254"
My ec2-local-instance is loging something like this: time="2020-04-28T20:49:49Z" level=error msg="HTTP 500 - NoCredentialProviders: no valid providers in chain\ncaused by: EnvAccessKeyNotFound: failed to find credentials in the environment.\nSharedCredsLoad: failed to load profile, default.\nEC2RoleRequestError: no EC2 instance role found\ncaused by: RequestError: send request failed\ncaused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
If I go inside the container, I can list all files inside /home/.aws
I didn't have a chance to test outside the proxy to see if it is related.
My team is experiencing the same issue with AWS SSO, only the local-container-endpoints process cannot parse the profile correctly with the sso_* configuration options, but it works correctly when we rely on our workaround via credential_process. The AWS CLI v1 and v2 do not experience this same limitation.
Currently, I am attempting to update the compiled AWS SDK version with the binary here but am not successful at resolving the issue (even with a fully update SDK, and the same behavior remains)
After recompiling with Debug logging enabled, we never get past this point in the temp credential processing. I'll be digging deeper to understand how the internal session is being configured as it appears the failure is within this internal handling.
UPDATE: In my case it seems this might have been a transient issue. After reboot it seemed to work (I thought that only happened to macOS & Windows !?!?). I'll update here again if this is not the case. Am on ArchLinux with everything up-to-date, though there were no udpates to any of the tooling involved in this.
Similar issue here. I have the following in the docker-compose:
environment:
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: /creds
ECS_CONTAINER_METADATA_URI: http://169.254.170.2/v3
When I try to curl that endpoint from within the aws-ecs-local-containers-endpoints container:
# curl -v 169.254.170.2/v3/creds
* Trying 169.254.170.2...
* TCP_NODELAY set
* Connected to 169.254.170.2 (169.254.170.2) port 80 (#0)
> GET /v3/creds HTTP/1.1
> Host: 169.254.170.2
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Fri, 11 Sep 2020 15:39:23 GMT
< Content-Length: 19
<
404 page not found
* Connection #0 to host 169.254.170.2 left intact
If I take out the v3/ from the URL I get:
# curl -v 169.254.170.2/creds
* Trying 169.254.170.2...
* TCP_NODELAY set
* Connected to 169.254.170.2 (169.254.170.2) port 80 (#0)
> GET /creds HTTP/1.1
> Host: 169.254.170.2
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Fri, 11 Sep 2020 15:40:17 GMT
< Content-Length: 486
<
Internal Server Error: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, dev.
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
* Connection #0 to host 169.254.170.2 left intact
Other possibly relevant state of the amazon-ecs-local-container-endpoints container:
# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
60: eth0@if61: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:a9:fe:aa:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 169.254.170.2/24 brd 169.254.170.255 scope global eth0
valid_lft forever preferred_lft forever
# ss -plnt
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 4096 127.0.0.11:37043 0.0.0.0:*
LISTEN 0 4096 *:80 *:* users:(("local-container",pid=1,fd=3))
``
UPDATE: I wasn't curling with the /creds. Removing the "/v3" from the metadata URI and adding the relative URI (/creds or /role/role_name) did the trick for me.
Seeing the same issue here. When I curl http://169.254.170.2/v3 from the ecs-container-endpoints container though, I get Internal Server Error: Failed to list running containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?. I have /var/run mounted on the container as per the guide. Any idea what's up?
Seeing the same issue here. When I curl http://169.254.170.2/v3 from the ecs-container-endpoints container though, I get
Internal Server Error: Failed to list running containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?. I have /var/run mounted on the container as per the guide. Any idea what's up?
@jskrt This was happening for me with Docker Desktop for Mac version 3.x. Downgraded to 2.3.0.4 and curl http://169.254.170.2/v3 works
I am getting the same issue HTTP 500 - NoCredentialProviders: no valid providers in chain\ncaused by: EnvAccessKeyNotFound: failed to find credentials in the environment.\nSharedCredsLoad: failed to load profile, spm-dev.\nEC2RoleRequestError: no EC2 instance role found\ncaused by: RequestError: send request failed\ncaused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: connection refused, with AWS SSO can't work out why
I am getting the same issue HTTP 500 - NoCredentialProviders: no valid providers in chain\ncaused by: EnvAccessKeyNotFound: failed to find credentials in the environment.\nSharedCredsLoad: failed to load profile, spm-dev.\nEC2RoleRequestError: no EC2 instance role found\ncaused by: RequestError: send request failed\ncaused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: connection refused, with AWS SSO can't work out why
Found the problem now. This flat does not work with SSO. It only works when I copy the credentials including the session token into the credentials file. Will raise new ticket
Hi guys, sorry, missed the whole convo. I ended up using the env file and this works for the team. Didn't dig further.
I am struggling with this exact same error as described in this issue. I would really like to get a response from AWS on this. I am using the example compose file that could not be more simple
version: '3'
services:
app:
image: ruby:2.7-alpine
ports:
- 80:80
Yet when I do anything to utilize AWS services from within that container—such as using the ruby aws-sdk-s3 gem— I get:
Traceback (most recent call last):
16: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:179:in `apply_defaults'
15: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:191:in `resolve'
14: from /usr/local/lib/ruby/2.7.0/set.rb:328:in `each'
13: from /usr/local/lib/ruby/2.7.0/set.rb:328:in `each_key'
12: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:191:in `block in resolve'
11: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:207:in `value_at'
10: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:214:in `resolve_defaults'
9: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:59:in `each'
8: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:59:in `each'
7: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:215:in `block in resolve_defaults'
6: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:72:in `call'
5: from /usr/local/bundle/gems/aws-sdk-s3-1.91.0/lib/aws-sdk-s3/plugins/s3_signer.rb:14:in `block in <class:S3Signer>'
4: from /usr/local/bundle/gems/aws-sdk-s3-1.91.0/lib/aws-sdk-s3/plugins/s3_signer.rb:209:in `build_v4_signer'
3: from /usr/local/bundle/gems/aws-sdk-s3-1.91.0/lib/aws-sdk-s3/plugins/s3_signer.rb:209:in `new'
2: from /usr/local/bundle/gems/aws-sigv4-1.2.3/lib/aws-sigv4/signer.rb:124:in `initialize'
1: from /usr/local/bundle/gems/aws-sigv4-1.2.3/lib/aws-sigv4/signer.rb:621:in `extract_credentials_provider'
Aws::Sigv4::Errors::MissingCredentialsError (missing credentials, provide credentials with one of the following options:)
- :access_key_id and :secret_access_key
- :credentials
- :credentials_provider
and trying to curl the endpoint I receive:
bash-5.1# curl 169.254.170.2/creds
Internal Server Error: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, default.
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: connection refused
I have attached to the ecs container and double checked that my credentials are mounted in /home/.aws and found that they are there as expected. I have searched far and wide for a solution to this and found nothing. It seems the error lies in the Docker image amazon/amazon-ecs-local-container-endpoints
I have tried the V3 endpoint setup, V4 endpoint setup, and even the V2 endpoint setup and all of it results in the same behavior.
Please help.
@AHaymond Can you please add more info. Give us your entire compose file (I don't see the local endpoints container in what you posted).
I have attached to the ecs container and double checked that my credentials are mounted in /home/.aws and found that they are there as expected
Which container is this?
I am utilizing docker-compose.overrides.yml here it is:
version: "2"
networks:
# This special network is configured so that the local metadata
# service can bind to the specific IP address that ECS uses
# in production
credentials_network:
driver: bridge
ipam:
config:
- subnet: "169.254.170.0/24"
gateway: 169.254.170.1
services:
# This container vends credentials to your containers
ecs-local-endpoints:
# The Amazon ECS Local Container Endpoints Docker Image
image: amazon/amazon-ecs-local-container-endpoints
volumes:
# Mount /var/run so we can access docker.sock and talk to Docker
- /var/run:/var/run
# Mount the shared configuration directory, used by the AWS CLI and AWS SDKs
# On Windows, this directory can be found at "%UserProfile%\.aws"
- $HOME/.aws/:/home/.aws/
environment:
# define the home folder; credentials will be read from $HOME/.aws
HOME: "/home"
# You can change which AWS CLI Profile is used
AWS_PROFILE: "default"
networks:
credentials_network:
# This special IP address is recognized by the AWS SDKs and AWS CLI
ipv4_address: "169.254.170.2"
# Here we reference the application container that we are testing
# You can test multiple containers at a time, simply duplicate this section
# and customize it for each container, and give it a unique IP in 'credentials_network'.
app:
depends_on:
- ecs-local-endpoints
networks:
credentials_network:
ipv4_address: "169.254.170.3"
environment:
AWS_DEFAULT_REGION: "us-east-1"
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/creds"
ECS_CONTAINER_METADATA_URI: "http://169.254.170.2/v3"
I am using Docker Desktop for Mac OS 3.1.0 <<< could this be causing an issue? Is it a problem with the VM that Docker Desktop uses?
I'm using 3.1.0 on Mac OS... so I don't think that's it...
Hmmmm.
SharedCredsLoad: failed to load profile, default.
So this is the issue..
Is your profile named default? If not you need to specify AWS_DEFAULT_PROFILE on the local endpoints container. Something is wrong with the setup for your local creds in .aws
Yes it is default. I just tested it on Arch Linux with the same exact setup and had no issues retrieving credentials.
Interesting... how about trying to set AWS_SHARED_CREDENTIALS_FILE on the local endpoints container. Set that to the absolute container path where the credentials file can be found. The full path inside the container. Not the directory.
set the absolute path and got the same error
SharedCredsLoad: failed to load profile, default.
So local container endpoints works for me, and as you noted, it works on Arch Linux. I don't think there's a bug in the container. There's likely something with your setup on Mac that isn't working. I'm sorry but you'll probably have to figure that out yourself- the above are the only ideas I can think of to try.
I am struggling with this exact same error as described in this issue. I would really like to get a response from AWS on this. I am using the example compose file that could not be more simple
version: '3' services: app: image: ruby:2.7-alpine ports: - 80:80Yet when I do anything to utilize AWS services from within that container—such as using the ruby aws-sdk-s3 gem— I get:
Traceback (most recent call last): 16: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:179:in `apply_defaults' 15: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:191:in `resolve' 14: from /usr/local/lib/ruby/2.7.0/set.rb:328:in `each' 13: from /usr/local/lib/ruby/2.7.0/set.rb:328:in `each_key' 12: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:191:in `block in resolve' 11: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:207:in `value_at' 10: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:214:in `resolve_defaults' 9: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:59:in `each' 8: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:59:in `each' 7: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:215:in `block in resolve_defaults' 6: from /usr/local/bundle/gems/aws-sdk-core-3.112.1/lib/seahorse/client/configuration.rb:72:in `call' 5: from /usr/local/bundle/gems/aws-sdk-s3-1.91.0/lib/aws-sdk-s3/plugins/s3_signer.rb:14:in `block in <class:S3Signer>' 4: from /usr/local/bundle/gems/aws-sdk-s3-1.91.0/lib/aws-sdk-s3/plugins/s3_signer.rb:209:in `build_v4_signer' 3: from /usr/local/bundle/gems/aws-sdk-s3-1.91.0/lib/aws-sdk-s3/plugins/s3_signer.rb:209:in `new' 2: from /usr/local/bundle/gems/aws-sigv4-1.2.3/lib/aws-sigv4/signer.rb:124:in `initialize' 1: from /usr/local/bundle/gems/aws-sigv4-1.2.3/lib/aws-sigv4/signer.rb:621:in `extract_credentials_provider' Aws::Sigv4::Errors::MissingCredentialsError (missing credentials, provide credentials with one of the following options:) - :access_key_id and :secret_access_key - :credentials - :credentials_providerand trying to curl the endpoint I receive:
bash-5.1# curl 169.254.170.2/creds Internal Server Error: NoCredentialProviders: no valid providers in chain caused by: EnvAccessKeyNotFound: failed to find credentials in the environment. SharedCredsLoad: failed to load profile, default. EC2RoleRequestError: no EC2 instance role found caused by: RequestError: send request failed caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: connection refusedI have attached to the ecs container and double checked that my credentials are mounted in
/home/.awsand found that they are there as expected. I have searched far and wide for a solution to this and found nothing. It seems the error lies in the Docker imageamazon/amazon-ecs-local-container-endpointsI have tried the V3 endpoint setup, V4 endpoint setup, and even the V2 endpoint setup and all of it results in the same behavior.
Please help.
The solution for me was to clone the repo up the version of aws-sdk-go to the latest version, go mod vendor and recompile works perfectly now with SSO. May be this will fix your issue too.
What I don't quite understand is why this container calls out to the EC2 Instance Metadata Service. That can't really work unless you run it in EC2, or mock that endpoint too.
time="2021-04-05T12:44:49Z" level=info msg="ecs-local-container-endpoints 1.3.0 (4fa3c29) ECS Agent 1.27.0 compatible"
time="2021-04-05T12:44:49Z" level=info msg=Running...
time="2021-04-05T12:44:59Z" level=error msg="HTTP 500 - NoCredentialProviders: no valid providers in chain\ncaused by: EnvAccessKeyNotFound: failed to find credentials in the environment.\nSharedCredsLoad: failed to load profile, default.\nEC2RoleRequestError: no EC2 instance role found\ncaused by: RequestError: send request failed\ncaused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: connection refused"
I ran into this today as well. I manged to figure out my issue, hopefully this helps someone. For me it was a misunderstanding of how this container uses credentials from ~/.aws/. I assumed it could use the profiles created by aws configure sso, but doesn't appear to be the case. It seems to only work using the profiles in ~/.aws/credentials which should look like this:
[docker_compose_profile]
aws_access_key_id=YOUR_ACCESS_KEY_ID
aws_secret_access_key=YOUR_SECRET_ACCESS_KEY
aws_session_token=YOUR_SESSION_TOKEN
You can get those values from the sso web portal by clicking the "Command line or programmatic access". Keep in mind you need the aws_session_token if you have MFA enabled (https://aws.amazon.com/premiumsupport/knowledge-center/authenticate-mfa-cli/)
I like @davidsteed's solution above... seems like the most robust BUT I don't understand why the latest version of the go sdk isn't already incorporated into the container. (It's been a couple years and IDC creds still don't seem to work or are not documented...) Docs I've read:
- https://aws.amazon.com/blogs/compute/a-guide-to-locally-testing-containers-with-amazon-ecs-local-endpoints-and-docker-compose/
- https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html
- https://aws.github.io/aws-sdk-go-v2/docs/
- https://github.com/awslabs/amazon-ecs-local-container-endpoints/blob/mainline/docs/configuration.md (seems to indicate use of aws sdk go v1 :( )
Can anyone maintaining this repo answer that please?
Another wtg for compatibility with Identity Center/AWS SSO is to parse one of the JSON files under ~/.aws/cli/cache (using jq for example) and set those values and env vars for the container.
I don't love this solution and wish that the ecs-local-container container could parse and use that file natively (or some other IDC mechanism... )
This looks promising but not sure how to implement: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html
@robotparty the goal of this project is to locally simulate task IAM roles.
looks like the code is updated: https://github.com/awslabs/amazon-ecs-local-container-endpoints/commits/mainline
@Lou1415926 Can we get a new release of this package?
EDIT: Sorry I see its on Docker Hub: https://hub.docker.com/r/amazon/amazon-ecs-local-container-endpoints/tags
Just no release notes. @robotparty can you try out the latest image on docker hub and see if it works for SSO?