Cannot assume roles with any version since 6.3.1

[bryanhorstmann]

• [x] I am using the latest release of AWS Vault
• [x] I have provided my .aws/config (redacted if necessary)
• [x] I have provided the debug output using aws-vault --debug (redacted if necessary)

I'm pulling my hair out trying to debug this. aws-vault works perfectly fine on 6.3.1. But if I update to any version afterward, I get the following error:

aws-vault: error: exec: Failed to get credentials for child: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 59d8c36c-bfe9-467a-ae51-e01de21e8c73, api error AccessDenied: User: arn:aws:iam::111111111111:user/bryan.h is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::222222222222:role/OrganizationAccountAccessRole-Admin

I have tested with 6.4.0, 6.5.0, 6.6.0.

This issue is occuring for multiple users across the business. The only thing we all have in common is we're running Apple Silicon chipset.

My .aws/config

→ cat .aws/config
[profile parent]
mfa_serial=arn:aws:iam::111111111111:mfa/bryan.h
region=us-east-1

[profile child]
mfa_serial=arn:aws:iam::111111111111:mfa/bryan.h
source_profile=parent
region=eu-west-1
role_arn=arn:aws:iam::222222222222:role/OrganizationAccountAccessRole-Admin

Testing

Versions

→ aws-vault6.3.1 --version
v6.3.1

→ aws-vault6.6.0 --version
v6.6.0

Not assuming roles

→ aws-vault6.3.1 exec parent -- aws s3 ls
2022-06-28 09:55:20 bucket1
2022-06-27 15:53:09 bucket2
2020-06-30 11:01:59 bucket3
2019-11-25 13:38:34 bucket4

→ aws-vault6.6.0 exec parent -- aws s3 ls
2022-06-28 09:55:20 bucket1
2022-06-27 15:53:09 bucket2
2020-06-30 11:01:59 bucket3
2019-11-25 13:38:34 bucket4

Assuming roles

→ aws-vault6.3.1 exec child -- aws s3 ls
2021-10-29 09:17:23 bucket1
2020-06-30 10:59:22 bucket2
2020-05-07 10:25:33 bucket3
2021-04-15 11:20:03 bucket4

→ aws-vault6.6.0 exec child -- aws s3 ls
aws-vault: error: exec: Failed to get credentials for child: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: fb49ba0d-f9b7-40a3-b76c-d167603a7b7a, api error AccessDenied: User: arn:aws:iam::111111111111:user/bryan.h is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::222222222222:role/OrganizationAccountAccessRole-Admin

Debug logs

→ aws-vault6.6.0 exec yoco-aws-dev --debug -- aws s3 ls
2022/07/15 11:04:47 aws-vault v6.6.0
2022/07/15 11:04:47 Loading config file /Users/bryanhorstmann/.aws/config
2022/07/15 11:04:47 Parsing config file /Users/bryanhorstmann/.aws/config
2022/07/15 11:04:47 [keyring] Considering backends: [keychain]
2022/07/15 11:04:47 Profile 'default' missing in config file
2022/07/15 11:04:47 Profile 'default' missing in config file
2022/07/15 11:04:47 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/07/15 11:04:47 [keyring] Found 3 results
2022/07/15 11:04:47 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/07/15 11:04:47 [keyring] Found 3 results
2022/07/15 11:04:47 profile parent: using stored credentials
2022/07/15 11:04:47 profile parent: using GetSessionToken (with MFA)
2022/07/15 11:04:47 profile child: using AssumeRole (chained MFA)
2022/07/15 11:04:47 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/07/15 11:04:47 [keyring] Found 3 results
2022/07/15 11:04:47 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/07/15 11:04:47 [keyring] Found 3 results
2022/07/15 11:04:47 [keyring] Querying keychain for service="aws-vault", account="sts.GetSessionToken,eW9jby1hd3MtbWFzdGVy,YXJuOmF3czppYW06OjA1MjI5NDI1NTQwMjptZmEvYnJ5YW4uaA,1657904023", keychain="aws-vault.keychain"
2022/07/15 11:04:47 [keyring] Found item "aws-vault session for parent (expires 2022-07-15T16:53:43Z)"
2022/07/15 11:04:47 Re-using cached credentials ****************I2DX from sts.GetSessionToken, expires in 7h48m55.437109s
aws-vault: error: exec: Failed to get credentials for child: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: c44e0fb9-8586-495d-ba6d-8e09eddf6325, api error AccessDenied: User: arn:aws:iam::111111111111:user/bryan.h is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::222222222222:role/OrganizationAccountAccessRole-Admin

[yorinasub17]

I had the same issue after upgrading, and was able to resolve this after adding a [default] profile with a region:

[default]
region = us-east-1

[bryanhorstmann]

Hi @yorinasub17,

I have added that and still have the same problem.

[mtibben]

The major change between 6.3.1 and 6.40 was the use of the v2 AWS SDK.

In your debug logs I note 2022/07/15 11:04:47 Re-using cached credentials, which means the v2 SDK is trying to re-use credentials generated by v1.

I would suggest firstly doing a aws-vault clear - you may find v6.4.0 works fine after doing this. If not, post the debug logs after doing the clear.

Secondly I would suggest trying this with aws-vault --debug exec --no-session

[bryanhorstmann]

Thanks for the feedback @mtibben. Still having issues unfortunately.

Order of events:

• aws-vault clear
• aws-vault6.3.1 exec child -- aws s3 ls works
• aws-vault clear just to confirm its clear
• aws-vault6.6.0 --debug exec child --no-session -- aws s3 ls fails

→ /usr/local/bin/aws-vault6.6.0 --debug exec child --no-session -- aws s3 ls
2022/09/02 14:18:04 aws-vault v6.6.0
2022/09/02 14:18:04 Loading config file /Users/bryanhorstmann/.aws/config
2022/09/02 14:18:04 Parsing config file /Users/bryanhorstmann/.aws/config
2022/09/02 14:18:04 [keyring] Considering backends: [keychain]
2022/09/02 14:18:04 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/02 14:18:04 [keyring] Found 3 results
2022/09/02 14:18:04 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/02 14:18:04 [keyring] Found 3 results
2022/09/02 14:18:04 profile parent: using stored credentials
2022/09/02 14:18:04 profile parent: skipping GetSessionToken because disabled
2022/09/02 14:18:04 profile child: using AssumeRole (with MFA)
2022/09/02 14:18:04 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/02 14:18:04 [keyring] Found 3 results
2022/09/02 14:18:04 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/02 14:18:04 [keyring] Found 3 results
2022/09/02 14:18:04 [keyring] Querying keychain for service="aws-vault", account="sts.AssumeRole,eW9jby1hd3MtZGV2,YXJuOmF3czppYW06OjA1MjI5NDI1NTQwMjptZmEvYnJ5YW4uaA,-62135596800", keychain="aws-vault.keychain"
2022/09/02 14:18:04 [keyring] No results found
Enter MFA code for arn:aws:iam::111111111111:mfa/bryan.h: 220360
2022/09/02 14:18:13 Looking up keyring for 'parent'
2022/09/02 14:18:13 [keyring] Querying keychain for service="aws-vault", account="parent", keychain="aws-vault.keychain"
2022/09/02 14:18:13 [keyring] Found item "aws-vault (parent)"
aws-vault: error: exec: Failed to get credentials for child: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 7780ab14-72d9-4746-99f4-b1cfac6e82e5, api error AccessDenied: User: arn:aws:iam::111111111111:user/bryan.h is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::222222222222:role/OrganizationAccountAccessRole-Admin

I even tried, running the clear command against all the binaries I have on my machine and tried again. Same result.

# bryanhorstmann at THVQ46M6RN in ~ [14:22:38]
→ aws-vault clear
Cleared 1 sessions.

# bryanhorstmann at THVQ46M6RN in ~ [14:23:35]
→ aws-vault6.3.1 clear
Cleared 0 sessions.

# bryanhorstmann at THVQ46M6RN in ~ [14:23:40]
→ aws-vault6.6.0 clear
Cleared 0 sessions.

# bryanhorstmann at THVQ46M6RN in ~ [14:23:49]
→ aws-vault6.6.0 exec child --debug --no-session -- aws s3 ls
2022/09/02 14:24:20 aws-vault v6.6.0
2022/09/02 14:24:20 Loading config file /Users/bryanhorstmann/.aws/config
2022/09/02 14:24:20 Parsing config file /Users/bryanhorstmann/.aws/config
2022/09/02 14:24:20 [keyring] Considering backends: [keychain]
2022/09/02 14:24:20 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/02 14:24:20 [keyring] Found 3 results
2022/09/02 14:24:20 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/02 14:24:20 [keyring] Found 3 results
2022/09/02 14:24:20 profile parent: using stored credentials
2022/09/02 14:24:20 profile parent: skipping GetSessionToken because disabled
2022/09/02 14:24:20 profile child: using AssumeRole (with MFA)
2022/09/02 14:24:20 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/02 14:24:20 [keyring] Found 3 results
2022/09/02 14:24:20 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/02 14:24:20 [keyring] Found 3 results
2022/09/02 14:24:20 [keyring] Querying keychain for service="aws-vault", account="sts.AssumeRole,eW9jby1hd3MtZGV2,YXJuOmF3czppYW06OjA1MjI5NDI1NTQwMjptZmEvYnJ5YW4uaA,-62135596800", keychain="aws-vault.keychain"
2022/09/02 14:24:20 [keyring] No results found
Enter MFA code for arn:aws:iam::111111111111:mfa/bryan.h: 538170
2022/09/02 14:24:23 Looking up keyring for 'parent'
2022/09/02 14:24:23 [keyring] Querying keychain for service="aws-vault", account="parent", keychain="aws-vault.keychain"
2022/09/02 14:24:23 [keyring] Found item "aws-vault (parent)"
aws-vault: error: exec: Failed to get credentials for child: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 6b35cb7e-aad7-4e90-aa1f-d97944fc676a, api error AccessDenied: User: arn:aws:iam::111111111111:user/bryan.h is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::222222222222:role/OrganizationAccountAccessRole-Admin

# bryanhorstmann at THVQ46M6RN in ~ [14:24:24]
→

[mtibben]

can you provide the debug output for the successful aws-vault6.3.1 exec

[bryanhorstmann]

Apologies for the delayed response, this was one of those "saw notification on phone and read it then never actioned" events.

→ aws-vault6.3.1 exec --debug child --no-session -- aws s3 ls
2022/09/06 17:50:58 aws-vault v6.3.1
2022/09/06 17:50:58 Loading config file /Users/bryanhorstmann/.aws/config
2022/09/06 17:50:58 Parsing config file /Users/bryanhorstmann/.aws/config
2022/09/06 17:50:58 [keyring] Considering backends: [keychain]
2022/09/06 17:50:58 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/06 17:50:58 [keyring] Found 3 results
2022/09/06 17:50:58 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/06 17:50:58 [keyring] Found 3 results
2022/09/06 17:50:58 profile parent: using stored credentials
2022/09/06 17:50:58 profile parent: skipping GetSessionToken because disabled
2022/09/06 17:50:58 profile child: using AssumeRole (with MFA)
2022/09/06 17:50:58 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/06 17:50:58 [keyring] Found 3 results
2022/09/06 17:50:58 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/06 17:50:58 [keyring] Found 3 results
2022/09/06 17:50:58 [keyring] Querying keychain for service="aws-vault", account="sts.AssumeRole,eW9jby1hd3MtZGV2,YXJuOmF3czppYW06OjA1MjI5NDI1NTQwMjptZmEvYnJ5YW4uaA,-62135596800", keychain="aws-vault.keychain"
2022/09/06 17:50:58 [keyring] No results found
Enter token for arn:aws:iam::111111111111:mfa/bryan.h: 564244
2022/09/06 17:51:05 Using STS endpoint https://sts.amazonaws.com
2022/09/06 17:51:05 Looking up keyring for 'parent'
2022/09/06 17:51:05 [keyring] Querying keychain for service="aws-vault", account="parent", keychain="aws-vault.keychain"
2022/09/06 17:51:05 [keyring] Found item "aws-vault (parent)"
2022/09/06 17:51:06 Generated credentials ****************XXXX using AssumeRole, expires in 1h0m0.274491s
2022/09/06 17:51:06 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/06 17:51:06 [keyring] Found 3 results
2022/09/06 17:51:06 [keyring] Querying keychain for service="aws-vault", keychain="aws-vault.keychain"
2022/09/06 17:51:06 [keyring] Found 3 results
2022/09/06 17:51:06 [keyring] Checking keychain status
2022/09/06 17:51:06 [keyring] Keychain status returned nil, keychain exists
2022/09/06 17:51:06 [keyring] Keychain item trusts keyring
2022/09/06 17:51:06 [keyring] Adding service="aws-vault", label="aws-vault session for child (expires 2022-09-06T16:51:07Z)", account="sts.AssumeRole,eW9jby1hd3MtZGV2,YXJuOmF3czppYW06OjA1MjI5NDI1NTQwMjptZmEvYnJ5YW4uaA,1662483067", trusted=true to osx keychain "aws-vault.keychain"
2022/09/06 17:51:06 Setting subprocess env: AWS_DEFAULT_REGION=eu-west-1, AWS_REGION=eu-west-1
2022/09/06 17:51:06 Setting subprocess env: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
2022/09/06 17:51:06 Setting subprocess env: AWS_SESSION_TOKEN, AWS_SECURITY_TOKEN
2022/09/06 17:51:06 Setting subprocess env: AWS_SESSION_EXPIRATION
2022/09/06 17:51:06 Exec command aws s3 ls
2022/09/06 17:51:06 Found executable /usr/local/bin/aws
2021-10-29 09:17:23 bucket1
2020-06-30 10:59:22 bucket2
2020-05-07 10:25:33 bucket3
2021-04-15 11:20:03 bucket4

[bryanhorstmann]

Hi @mtibben, just following up on this. Any ideas or additional troubleshooting I can try?

[bryanhorstmann]

Hi @mtibben, just following up on this for guidance

[mtibben]

@bryanhorstmann I note you are using 2 different regions between the parent and child profiles. Perhaps try using the same region?

Can you also trying using the latest v7

[bryanhorstmann]

Hi @mtibben ,

Apologies for the delayed response. We figured this out by complete accident. As a team, we realised that it was only members of our Administrator group that could not assume roles. This group had no explicit assume role policy assigned, but was relying on blanket wildcard permissions to work. Once we assigned relevant assume role policies to the Administrator group it worked. This includes the latest v7.

Thank you for your assistance.

[sladesamuel]

For anyone else who may stumble across this thread with the same error, my issue was that in my config, I hadn't added the mfa_serial argument to both the parent and child profiles. I had assumed that using source_profile would inherit the MFA argument too.

$ cat ~/.aws/config
[default]
region=eu-west-2
output=json

[profile basic]
mfa_serial=XXXX

[profile admin]
source_profile=basic
role_arn=arn:aws:iam::123456789012:role/admin

:arrow_up: that failed, until I added the same mfa_serial line after role_arn under the admin profile.

[vietanhtran16]

For anyone else who may stumble across this thread with the same error, my issue was that in my config, I hadn't added the mfa_serial argument to both the parent and child profiles. I had assumed that using source_profile would inherit the MFA argument too.

$ cat ~/.aws/config
[default]
region=eu-west-2
output=json

[profile basic]
mfa_serial=XXXX

[profile admin]
source_profile=basic
role_arn=arn:aws:iam::123456789012:role/admin

⬆️ that failed, until I added the same mfa_serial line after role_arn under the admin profile.

Thanks for this. I was assuming the same thing and kept getting 403