corymhall
Unable to assume CDK lookup role using temporary session credentials
Having a bit of trouble getting this action to work. Using the following configuration:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v2
with:
aws-region: ${{ vars.AWS_REGION }}
role-to-assume: ${{ secrets.ROLE_TO_ASSUME }}
role-session-name: ci-session
- name: Synth
working-directory: apps/cdk
run: pnpm cdk synth
- name: Diff
uses: corymhall/cdk-diff-action@v1
with:
failOnDestructiveChanges: false
cdkOutDir: apps/cdk/cdk.out
githubToken: ${{ secrets.GITHUB_TOKEN }}
I get the following error:
AccessDenied: User: arn:aws:sts::***:assumed-role/github-action/ci-session is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region}
Seems like the issue is that AWS::AccountId and AWS::Region isn't templating correctly when assuming the CDK lookup role? Not exactly sure why that would be the case. I've included the full error logs, happy to provide additional information needed.
Full error log
Error performing stack diff: AccessDenied: User: arn:aws:sts::***:assumed-role/github-action/monorepoCISession is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region}
at throwDefaultError (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/smithy-client/dist-cjs/index.js:838:1)
Error: User: arn:aws:sts::***:assumed-role/github-action/monorepoCISession is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region}
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/smithy-client/dist-cjs/index.js:847:1
at de_CommandError (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/client-sts/dist-cjs/index.js:478:1)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:1
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/core/dist-cjs/index.js:165:1
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/middleware-retry/dist-cjs/index.js:3[20](https://github.com/<redacted>/monorepo/actions/runs/9616955847/job/26527645521?pr=854#step:6:21):1
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:1
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/credential-providers/dist-cjs/fromTemporaryCredentials.js:47:1
at coalesceProvider (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/core/dist-cjs/index.js:288:1) {
'$fault': 'client',
'$metadata': {
httpStatusCode: 403,
requestId: '3092bae5-14f9-49e6-84bc-ce1ae49[22](https://github.com/<redacted>/monorepo/actions/runs/9616955847/job/26527645521?pr=854#step:6:23)3d8',
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
},
Type: 'Sender',
Code: 'AccessDenied'
}
Error processing stages: AccessDenied: User: arn:aws:sts::***:assumed-role/github-action/monorepoCISession is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region}
at throwDefaultError (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/smithy-client/dist-cjs/index.js:838:1)
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/smithy-client/dist-cjs/index.js:847:1
at de_CommandError (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/client-sts/dist-cjs/index.js:478:1)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:1
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/core/dist-cjs/index.js:165:1
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:1
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:1
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/credential-providers/dist-cjs/fromTemporaryCredentials.js:47:1
at coalesceProvider (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/core/dist-cjs/index.js:288:1) {
'$fault': 'client',
'$metadata': {
httpStatusCode: 403,
requestId: '3092bae5-14f9-49e6-84bc-ce1ae492[23](https://github.com/<redacted>/monorepo/actions/runs/9616955847/job/26527645521?pr=854#step:6:24)d8',
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
},
Type: 'Sender',
Code: 'AccessDenied'
}
Error running process stages: AccessDenied: User: arn:aws:sts::***:assumed-role/github-action/monorepoCISession is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region}
at throwDefaultError (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/smithy-client/dist-cjs/index.js:838:1)
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/smithy-client/dist-cjs/index.js:847:1
at de_CommandError (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/client-sts/dist-cjs/index.js:478:1)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:1
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/core/dist-cjs/index.js:165:1
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:1
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:1
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/credential-providers/dist-cjs/fromTemporaryCredentials.js:47:1
at coalesceProvider (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/core/dist-cjs/index.js:[28](https://github.com/<redacted>/monorepo/actions/runs/9616955847/job/26527645521?pr=854#step:6:29)8:1) {
'$fault': 'client',
'$metadata': {
httpStatusCode: 403,
requestId: '3092bae5-14f9-49e6-84bc-ce1ae49223d8',
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
},
Type: 'Sender',
Code: 'AccessDenied'
}
Error performing diff: AccessDenied: User: arn:aws:sts::***:assumed-role/github-action/monorepoCISession is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region}
at throwDefaultError (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/smithy-client/dist-cjs/index.js:838:1)
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/smithy-client/dist-cjs/index.js:847:1
at de_CommandError (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/client-sts/dist-cjs/index.js:478:1)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:1
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/core/dist-cjs/index.js:165:1
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:1
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:34:1
at /home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@aws-sdk/credential-providers/dist-cjs/fromTemporaryCredentials.js:47:1
at coalesceProvider (/home/runner/work/_actions/corymhall/cdk-diff-action/v1/node_modules/@smithy/core/dist-cjs/index.js:288:1) {
'$fault': 'client',
'$metadata': {
httpStatusCode: 403,
requestId: '[30](https://github.com/<redacted>/monorepo/actions/runs/9616955847/job/26527645521?pr=854#step:6:31)92bae5-14f9-49e6-84bc-ce1ae49223d8',
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
},
Type: 'Sender',
Code: 'AccessDenied'
}
Turns out the issue is that AWS::AccountId and AWS::Region won't have an actual value in the CDK assembly unless we explicitly pass a value to props.env on all our stacks during synthesis. The action worked after env was provided.
@tranhl sorry you ran into this error, we should probably update the readme to note that you have to specify the env for this action to work, otherwise there is no way for it to know which account/region is the target.
@corymhall I think I stumbled on this as well but I'm not sure that I agree with the solution. All my stacks are working fine on the local console using AWS_PROFILE= and AWS SSO temporary credentials, the cdk synth command generates a generic manifest, that is environment-agnostic:
cat cdk.out/manifest.json| grep acc
"environment": "aws://unknown-account/unknown-region",
"environment": "aws://unknown-account/unknown-region",
"environment": "aws://unknown-account/unknown-region",
This is in-line with the actual documentation of the behaviour from CDK - example here. I do not need/want to make my stacks non-account agnostic.
Github Action
Then in my GH action setup, I do have an IAM Role that is successfully assumed via OIDC:
Run aws-actions/configure-aws-credentials@v4
Assuming role with OIDC
Authenticated as assumedRoleId AROAZ4SRUYMZEW:github-infra-cdk-diff
and I have granted this role the CloudFormationReadOnlyAccess AWS Managed policy that should be enough to perform the diff.
However the action is trying to assume the lookup role instead of the temporary credentials obtained via OIDC:
Error performing stack diff: AccessDenied: User: arn:aws:sts::xxx:assumed-role/githubActionsDeployRole/github-infra-cdk-diff is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region}
I would say the logic here should be exactly the opposite, use the temporary credentials if they are present (or maybe that could be a flag?)
EDIT: Before running this action, I tried to run cdk diff in the workflow and even though it runs without any error, it also shows the same problem of
current credentials could not be used to assume 'arn:aws:iam::xxx:role/cdk-hnb659fds-lookup-role-xxx-us-west-2', but are for the right account. Proceeding anyway.
I'll keep investigating why this does work on a local CLI with agnostic stacks but doesn't work with temporary credentials on the handcrafted IAM role.
@rantoniuk this guide here has some additional details about how the CDK gets credentials for things. The CDK CLI (and this action) will only use your credentials to assume-role into the role that actually has the permissions to perform the action.
https://github.com/aws/aws-cdk/wiki/Security-And-Safety-Dev-Guide#defaultstacksynthesizer
I have fixed the problem of the manual cdk diff in the pipeline, by adding the required IAM sts:AssumeRole permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::*:role/cdk-hnb659fds-*"
}
]
}
to the role that is assumed in the OIDC step:
- name: Authenticate Via OIDC Role
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
role-duration-seconds: 900
role-to-assume: ${{ vars.AWS_OIDC_IAM_DEPLOY_ROLE }}
role-session-name: github-infra-cdk-diff
and now I see exactly the same output as when I'm doing a diff locally on the CLI console.
Yet, in the next step, this action still has the same issue as before, trying to assume a role without expanding the AccountID placeholder:
Error performing stack diff: AccessDenied: User: arn:aws:sts::679849022850:assumed-role/githubActionsDeployRole/github-infra-cdk-diff is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::${AWS::AccountId}:role/cdk-hnb659fds-lookup-role-${AWS::AccountId}-${AWS::Region}
I'm going to try to build a custom version of the action with the swapped logic to see if that fixes the issue.
@corymhall the action works flawlessly when I removed the custom credentials logic and disabled associated tests and tested via
uses: rantoniuk/corymhall-cdk-diff-action@use-default-credentials
In the commented PR I see wrong warning:
This stack has an unknown environment which may mean the diff is performed against the wrong environment
that is also coming from the action logic. Take a look at the changes and let me know if you think we could incorporate them into your action. I think that should be the default behaviour but I'd be also happy with a flag.
Happy to provide a PR!
@rantoniuk thanks for digging into this, I think I'm finally seeing what the issue is. There are two cases that are hard to tell apart and we currently only support one of them
- User has stacks and each stack deploys to a different environment
- User has stacks and they all deploy to the same environment (not handling this one)
In the first case the user has to provide the environment otherwise there is no way to tell which environment to perform the diff for.
We just need to handle the second case. When we get the credentials here:
https://github.com/corymhall/cdk-diff-action/blob/94b25b348cc388533e137d572bdbe76cfe93f503/src/diff.ts?plain=1#L75-L87
If the user provides the lookup role and the arn has an unknown account/region we should just substitute the current account/region (from the current credentials) like we do for the partition. We should also still log a warning to let the user know that we are making this assumption because there is a chance that they should have provided the environment, but have a misconfiguration.
I won't be able to pick this one up in the short term, but would definitely take a PR!
@corymhall I agree about the unsupported case, but I fail to understand why would this action need to support anything at all instead of relying on the credentials provided in earlier pipeline steps.
The purpose of this action could/should be just to interpret the diff based on cdk synth's output that is done in earlier pipeline steps. That being said, it should not need to mingle with any credentials and even if the action itself would like to run the diff itself via the DefaultSynthesizer, then it should still rely on properly configured credentials before.
Whether the stack is environment-agnostic or not, is not to be decided by this action, but instead by the Stack configuration.
The reason my version works is because:
- my stacks are environment agnostic, meaning that the IaaC is universal and I'm deploying it to many environments
- I have already assumed in the pipeline a dedicated role:
- name: Authenticate Via OIDC Role
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
role-duration-seconds: 900
role-to-assume: ${{ vars.AWS_OIDC_IAM_DEPLOY_ROLE }}
role-session-name: github-infra-cdk-diff
- and that the
${{ vars.AWS_OIDC_IAM_DEPLOY_ROLE }}role has the necessary permission:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::*:role/cdk-hnb659fds-*"
}
]
}
to assume the corresponding lookup roles in the target account that are defined in the manifest.json output. (Here the target account is indeed the same account but that does not matter)
In other words, responding to your comment - I'm not providing any lookup roles explicitly, everything is handled by CDK itself.
For the 1) supported case User has stacks and each stack deploys to a different environment. In such case, the account configuration comes from either env configuration or from making the stack account-specific by hardcoding the account numbers into the stack.
My underlying logic here is: as long as cdk diff works as a manual step somewhere earlier in the pipeline, this action should work as well without any additional logic.
The purpose of this action could/should be just to interpret the diff based on cdk synth's output that is done in earlier pipeline steps. That being said, it should not need to mingle with any credentials and even if the action itself would like to run the diff itself via the DefaultSynthesizer, then it should still rely on properly configured credentials before.
Whether the stack is environment-agnostic or not, is not to be decided by this action, but instead by the Stack configuration.
I think we are saying similar things. This action is trying to mimic the behavior of the CDK CLI. When you run cdk diff from the CLI, the CLI will read the CloudAssembly (cdk.out) and if you are using the DefaultStackSynthesizer it will assume the necessary role in the target account to perform the diff. When you run cdk diff it will not just use whatever credentials you have configured locally, it will always assume the specified roles in the CloudAssembly.
This action attempts to do the same thing. When it performs the diff it reads the CloudAssembly and if you are using the DefaultStackSynthesizer then it will assume the specified role to perform the diff.
The reason you are running into this error in this action, but not when using the CLI is because the CLI just has more robust credential logic and this action suffers from the bug I mentioned in my last comment.
This action is trying to mimic the behavior of the CDK CLI
Is there any specific reason it does that instead of relying on CDK buil-in capabilities? After all, you anyway are running npx cdk synth before via CLI to create the CloudAssembly.
The reason you are running into this error in this action, but not when using the CLI is because the CLI just has more robust credential logic and this action suffers from the bug I mentioned in my last comment.
Again, from my perspective the bug is that it implements the custom logic in the first place, because when this is removed, everything works fine with the role that was assumed in the previous step and to which the short-lived credentials are present.
@rantoniuk the custom logic exists because I want more control over the output. Because of the custom logic this action is able to do things like highlight resources that have destructive changes, fail the workflow if there are destructive changes, filter on what resources to allow destructive changes on, etc.
If all you want is the raw diff output from cdk diff this action is probably not needed and something simpler would suffice.
Unfortunately this does not answer my question at all. What you're describing is post-authentication logic and this whole discussion is about why the custom logic exists for authentication: https://github.com/corymhall/cdk-diff-action/blob/773b5ec3d64f5d871e83a4408f9d9d9d24c6bc94/src/diff.ts#L72-L83
I think I explained my view above already and I don't want to pollute the issue. From my POV if you are fine with adding a flag to use the default credentials, then I'm happy to do it, but I don't see any point of implementing yet more complex custom logic to cover the use case in the action itself, because it shouldn't be here imo.
@rantoniuk
When you perform cdk diff this is roughly what is happening.
There is no ability to pick and choose which things you want and which you do not. In my case for this action, I want to change the very last thing. I want to take the output of fullDiff and do something else with it. In order to do that I have to replace the entire box with custom logic, and this includes the authentication piece.
This is probably more information than you care for, but it's why I don't want to add the flag that you are suggesting. Anyone should definitely feel free to fork this action and customize it to work better for you individual use case.
I just created a new v2-beta release which uses the @aws-cdk/toolkit-lib library to perform the diff and it should fix this issue. Please give it a try and let me know!
I just tested it and it seems it works flawlessly so I will now be able to drop my custom version when this gets released - great work!
- uses: actions/setup-node@v4
with:
cache-dependency-path: package-lock.json
node-version: 22
cache: 'npm'
- name: Synth stacks
run: npm install && npx cdk synth
- name: CDK diff
uses: corymhall/cdk-diff-action@v2-beta
with:
githubToken: ${{ steps.ghactions-auth.outputs.token }}
cdkOutDir: ${{ github.workspace }}/cdk.out
failOnDestructiveChanges: false
[12:30:52] Starting Diff ... [12:30:52] Retrieved account ID 123456789012 from disk cache [12:30:52] Retrieved account ID 123456789012 from disk cache [12:30:52] Assuming role 'arn:aws:iam::123456789012:role/cdk-hnb659fds-lookup-role-123456789012-us-west-2'. [12:30:53] [sdk info] STS.AssumeRole({"RoleArn":"arn:aws:iam::123456789012:role/cdk-hnb659fds-lookup-role-123456789012-us-west-2","RoleSessionName":"aws-cdk-runner"}) -> OK
I just tried with the new beta with the following input:
uses: corymhall/cdk-diff-action@v2-beta
with:
githubToken: ${{ secrets.GITHUB_TOKEN }}
cdkOutDir: ./infra/cdk.out
failOnDestructiveChanges: false
stackSelectorPatterns: '!Prod/*'
And getting the following error, though it is referring to our Prod stack account even though I have filtered it out:
Error running process stages: ToolkitError: Context lookups have been disabled. Make sure all necessary context is already in 'cdk.context.json' by running 'cdk synth' on a machine with sufficient AWS credentials and committing the result. Missing context keys: 'availability-zones:account=371488075649:region=ap-southeast-2, vpc-provider:account=371488075649:filter.tag:Name=main:region=ap-southeast-2:returnAsymmetricSubnets=true, hosted-zone:account=371488075649:domainName=******:region=ap-southeast-2'
For reference our single cdk file deploys to both accounts via the use of stages. In reading your commend:
In the first case the user has to provide the environment otherwise there is no way to tell which environment to perform the diff for.
Is there somewhere else the action should be notified of this?
@adamlyka can you provide more details on your setup along with any additional log messages? I've tried to reproduce the issue, but haven't been able to. How are you synthesizing the app in the action?
@corymhall I am doing the following synth prior:
cdk synth Dev/*
Which as a result, there are no attributes in the cdk.context.json which contain any production details, so the error is valid. Though the goal is to only have a diff on those items
@adamlyka thanks, I've been able to reproduce the issue now and tracked it down to a bug in the cdk library I'm using. I've created a PR upstream to fix it https://github.com/aws/aws-cdk-cli/pull/611