Unable to use IAM role as credential in CI agent (Gitlab)

[houdinisparks]

Hello!

Facing an error when trying to deploy to an ECS cluster using AWS Copilot. Logs below:

$ aws sts get-caller-identity
{
    "UserId": ".....",
    "Account": ".....",
    "Arn": "arn:aws:sts::...."
}

$ copilot --version
copilot version: v1.21.0

$ copilot svc deploy --name locust-master --env mgmt
It looks like your credential settings are misconfigured or missing:
https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials
- We recommend including your credentials in the shared credentials file.
- Alternatively, you can also set credentials through 
	* Environment Variables
	* EC2 Instance Metadata (credentials only)
More information: https://aws.github.io/copilot-cli/docs/credentials/
✘ RequestCanceled: request context canceled
caused by: context deadline exceeded
Cleaning up project directory and file based variables

The EC2 is attached to an IAM role (as seen aws sts get-caller-identity, however copilot is unable to retrieve the credentials.

According to the docs (https://aws.github.io/copilot-cli/docs/credentials/#application-credentials), copilot will use the default provider chain which will call the EC2 instance metadata endpoint to retrieve the IAM credentials automatically, not sure what is going wrong here.

Any help? Thanks!

[paragbhingre]

@houdinisparks I am wondering if you have default profile set in the ~/.aws/credentials file or AWS_PROFILE environment variable?

[houdinisparks]

@paragbhingre Ahh nope, we did not configure named profiles for our agent. Thought we can rely on the default provider chain to use the IAM credentials automatically.

Is named profile / the only mode supported for CoPilot?

[paragbhingre]

@houdinisparks quick question, when you call aws sts get-caller-identity how are you setting credentials i.e. accessKeyId and secretKey in order to get the details of the caller back?

[houdinisparks]

@paragbhingre nope, we did not specifically set anything. The AWS CLI should retrieve the IAM role permissions associated with the EC2 automatically

[paragbhingre]

@houdinisparks I tried reproducing it on my EC2 instance but I am able to deploy my services successfully without configuring any credentials except attached IAM role to my EC2 instance. Are you able to do copilot init? Is it erroring out on all the copilot commands?

[houdinisparks]

@paragbhingre hi! Nope I am unable to do a copilot init, it gives the same error below:

Curious, what are your IAM permissions?

[houdinisparks]

Hi @paragbhingre, we tried using a default profile also, but to no avail

$ mkdir ~/.aws

$ cp runner_aws_config.ini ~/.aws/config

$ cat ~/.aws/config
[default]
role_arn = arn:aws:iam::<account_id>:role/gitlab-non-gen-mgmtmzna-deploy-docker-machine
credential_source = Ec2InstanceMetadata
region = ap-southeast-1

$ aws sts get-caller-identity
{
    "UserId": ":botocore-session-1662022033",
    "Account": "<account_id>",
    "Arn": "arn:aws:sts::<account_id>:assumed-role/gitlab-non-gen-mgmtmzna-deploy-docker-machine/botocore-session-1662022033"
}

$ export IMAGE_TAG=$(cat .build_image_tag)

$ copilot --version
copilot version: v1.21.0

$ copilot init
It looks like your credential settings are misconfigured or missing:
https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials
- We recommend including your credentials in the shared credentials file.
- Alternatively, you can also set credentials through 
	* Environment Variables
	* EC2 Instance Metadata (credentials only)
More information: https://aws.github.io/copilot-cli/docs/credentials/
✘ RequestCanceled: request context canceled
caused by: context deadline exceeded

[paragbhingre]

That is super weird, because when I try same steps as you did then I can successfully deploy my application. Could you please run aws configure list and show me your output. I just want to check if profile is set properly.

And to answer to your question "Curious, what are your IAM permissions?" - IAM permissions will come into picture when copilot will be able to detect one 1st. You will see something like "An error occurred (AccessDenied) when calling the AssumeRole operation:..." error if IAM permissions are not enough.

[houdinisparks]

@paragbhingre Ahhaha, so strange. Yup realised the IAM permissions isn't the problem.

We manually extracted the AWS access credentials from the EC2 IAM role into AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and this worked for us in the end:

#!/bin/bash

ROLENAME=$(curl http://169.254.169.254/latest/meta-data/iam/security-credentials/ -s)
KeyURL="http://169.254.169.254/latest/meta-data/iam/security-credentials/"$ROLENAME"/"
wget $KeyURL -q -O Iam.json
export AWS_ACCESS_KEY_ID=$(cat Iam.json | jq -r '.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(cat Iam.json | jq -r '.SecretAccessKey')
export AWS_SECURITY_TOKEN=$(cat Iam.json | jq -r '.Token')
export AWS_SESSION_TOKEN=$(cat Iam.json | jq -r '.Token')
export AWS_DEFAULT_REGION=ap-southeast-1
rm Iam.json -f

aws sts get-caller-identity
export IMAGE_TAG=$(cat .build_image_tag)
copilot --version
copilot svc deploy --name locust-worker --env mgmt

[github-actions[bot]]

This issue is stale because it has been open 60 days with no response activity, and is tagged with pending/question. Remove the stale label, add a comment, or this will be closed in 14 days.

[github-actions[bot]]

This issue is closed due to inactivity. Feel free to reopen the issue if you have any follow-ups!