ExpiredTokenException Issue: while running aws-nuke from the AWS Codebuild project

[rbalman]

I have been running the aws-nuke from my machine using permanent AWS credentials but as I transitioned from my machine to AWS Codebuild I have to use the temporary Credentials provided by the assume-role which expires. Even after the expiry, aws-nuke continuous to send the API request, rather than exit the process.

Also, I would like to know, what is the best way to refresh the AWS token with context to aws-nuke. I find it especially hard with aws-nuke because it doesn't show any failure of token expiry while exitting.

[svenwltr]

Hello @BalmanRawat, this is a good catch.

How exactly do your temporary credentials work? Are these sessions with AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN? Or is it a AWS profile with role_arn and source_profile?

[rbalman]

I am running aws-nuke in AWS CodeBuild Project. I was using AWS CodeBuild Service Role to provide access to aws-nuke and that didn't work, but right now I am using permanent credentials in CodeBuild Project.

[svenwltr]

Okay, this means I have to setup a CodeBuild Project to reproduce this issue.

[adamcousins]

Im running this in codebuild with the following snippet to setup the codebuild credentials file with a CLI profile BuildSpec snippet below

 pre_build:
   commands:
     - export AWS_NukeProfileName="aws-nuke-example"
     - echo "Setting AWS CLI Credentials to profile $AWS_NukeProfileName"   
     - creds=`curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI`   
     - aws_access_key_id=`echo $creds | jq .AccessKeyId | tr -d '"'`   
     - aws_secret_access_key=`echo $creds | jq .SecretAccessKey | tr -d '"'`   
     - aws_session_token=`echo $creds | jq .Token | tr -d '"'`   
     - aws configure set aws_access_key_id $aws_access_key_id --profile $AWS_NukeProfileName   
     - aws configure set aws_secret_access_key $aws_secret_access_key --profile $AWS_NukeProfileName   
     - aws configure set aws_session_token $aws_session_token --profile $AWS_NukeProfileName   

then when i run the executable: aws-nuke --profile $AWS_NukeProfileName --config $AWS_NukeConfigFile --force

If your getting a token expiry, is this running longer than 1hour? (session timeout)

[svenwltr]

Ah, I see. Looks easy to reproduce. I will take a look.

[rbalman]

yes, it runs for more than an hour, that is the whole point.

[adamcousins]

@BalmanRawat ~~You could extend the duration on the role upto 12 hours~~ ~~https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/~~ ~~Not ideal but a solution for now.~~

[rbalman]

Thanks, @adamcousins I have done some work around this, but I was hoping to have some form of refreshing the token when it gets expired. Unfortunately, I couldn't refresh the token because aws-nuke keeps running even though the token has expired.

[adamcousins]

@BalmanRawat I just hit this issue. What was your work around?

[adamcousins]

@BalmanRawat You could extend the duration on the role upto 12 hours https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/ Not ideal but a solution for now.

This is not possible as AWS CodeBuild will only assume the service role with a 1hour duration. You cant chain two roles together (with assume-role) with a longer expiry if the source role is less than to desired session duration.

[rbalman]

@BalmanRawat I just hit this issue. What was your work around?

@adamcousins I have created an API user with static credentials and used it in the CodeBuild. Eg.

TIMEOUT_SECONDS=$(($TIMEOUT_HOURS*3600))
aws sts assume-role --role-arn $ROLE_ARN --role-session-name account --duration-seconds "$TIMEOUT_SECONDS" --output json --query "Credentials" | tee cred.json
ACCESS_KEY_ID=$(cat cred.json |jq -r .AccessKeyId);
SECRET_ACCESS_KEY=$(cat cred.json |jq -r .SecretAccessKey);
SESSION_TOKEN=$(cat cred.json |jq -r .SessionToken);

[njtrettel]

https://github.com/rebuy-de/aws-nuke/pull/667 partially addresses this issue

it adds the ability to pass in an --assume-role-arn to the CLI which nuke will assume before initializing any AWS clients. as a result, due to the nature of the stscreds library (which essentially wraps the AssumeRoleProvider), the credentials are refreshed automatically during the next Credentials.Get call.

this doesn't directly solve the issue mentioned here, but you could alternatively give it the role to assume (--assume-role-arn) instead of assuming it yourself and then you would be utilizing the auto-refresh feature