cdk diff: when a custom stack name and qualifier are used, the diff action falls back to the CDKToolkit stack for templates larger than 50KiB

[drew-marumoto]

Describe the bug

You have your account bootstrapped with a custom stack name and custom qualifier

cdk bootstrap aws://<Account>/us-west-2 --toolkit-stack-name "custom-stack" --qualifier "abc1234"

You have "@aws-cdk/core:bootstrapQualifier": "abc1234" in your cdk.json

Your account does not have a bootstrap stack named CDKToolkit

When you perform a "cdk diff" and your synthesized template is larger than 50KiB the action will not upload your template to the asset bucket created by your custom stack and perform a propper diff by generating a changeset. Instead the action will fall back, and look for the default stack name of "CDKToolkit" which will not be found resulting in a diff that is not based on a changeset and only based upon a template comparison.

Additionally you will receive the following message

The environment aws://XXXXXXXX/us-west-2 doesn't have the CDK toolkit stack (CDKToolkit) installed. Use cdk bootstrap "aws://XXXXXXXXX/us-west-2" to setup your environment for use with the toolkit.
The template for stack "CdkLargeTemplateStack" is 78KiB. Templates larger than 50KiB must be uploaded to S3.
Run the following command in order to setup an S3 bucket in this environment, and then re-deploy:

cdk bootstrap aws://XXXXXXXXX/us-west-2

This behavior does not exist for templates less that 50KiB in size

Expected Behavior

The diff behavior should be the same regardless of template size and the diff action should utilize all resources of the stack specified by the qualifier.

Current Behavior

Perform a "cdk diff -vvv" of a stack with a synthesized template larger than 50KiB. You can see in the output that the action assumes the lookup and deployment roles of the custom stack and is able to lookup the bootstrap version of the custom stack.

[09:38:17] outdir: cdk.out
[09:38:17] env: {
  CDK_DEFAULT_REGION: 'us-west-2',
  CDK_DEFAULT_ACCOUNT: 'XXXXXXXXXX',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '36.0.0',
  CDK_CLI_VERSION: '2.128.0'
}
[09:38:17] Notices refreshed
Stack CdkLargeTemplateStack
[09:38:19] [trace] SdkProvider#resolveEnvironment()
[09:38:19] [trace]   SdkProvider#defaultAccount()
[09:38:19] [trace] SdkProvider#baseCredentialsPartition()
[09:38:19] [trace]   SdkProvider#resolveEnvironment()
[09:38:19] [trace]   SdkProvider#obtainBaseCredentials()
[09:38:19] [trace]     SdkProvider#defaultAccount()
[09:38:19] [trace]     SdkProvider#defaultCredentials()
[09:38:19] [trace]   SDK#currentAccount()
[09:38:19] [trace]     SDK#forceCredentialRetrieval()
[09:38:19] Retrieved account ID XXXXXXXXXXX from disk cache
[09:38:19] [trace] SdkProvider#forEnvironment()
[09:38:19] [trace]   SdkProvider#resolveEnvironment()
[09:38:19] [trace]   SdkProvider#obtainBaseCredentials()
[09:38:19] [trace]     SdkProvider#defaultAccount()
[09:38:19] [trace]     SdkProvider#defaultCredentials()
[09:38:19] [trace]   SdkProvider#withAssumedRole()
[09:38:19] Assuming role 'arn:aws:iam::XXXXXXXXXXX:role/cdk-abc1234-lookup-role-XXXXXXXXXX-us-west-2'.
[09:38:19] [trace]   SDK#forceCredentialRetrieval()
[09:38:19] [trace] SDK#ssm()
[09:38:19] [trace]   SDK#wrapServiceErrorHandling()
[09:38:20] [AWS ssm 200 0.445s 0 retries] getParameter({ Name: '/cdk-bootstrap/abc1234/version' })
[09:38:20] [trace] SDK#cloudFormation()
[09:38:20] [trace]   SDK#wrapServiceErrorHandling()
[09:38:21] [AWS cloudformation 200 0.76s 0 retries] describeStacks({ StackName: 'CdkLargeTemplateStack' })
[09:38:22] [AWS cloudformation 200 0.825s 0 retries] getTemplate({ StackName: 'CdkLargeTemplateStack', TemplateStage: 'Original' })
[09:38:22] [trace] SdkProvider#resolveEnvironment()
[09:38:22] [trace]   SdkProvider#defaultAccount()
[09:38:22] [trace] SdkProvider#resolveEnvironment()
[09:38:22] [trace]   SdkProvider#defaultAccount()
[09:38:22] [trace] SdkProvider#baseCredentialsPartition()
[09:38:22] [trace]   SdkProvider#resolveEnvironment()
[09:38:22] [trace]   SdkProvider#obtainBaseCredentials()
[09:38:22] [trace]     SdkProvider#defaultAccount()
[09:38:22] [trace]     SdkProvider#defaultCredentials()
[09:38:22] [trace]   SDK#currentAccount()
[09:38:22] [trace]     SDK#forceCredentialRetrieval()
[09:38:22] Retrieved account ID XXXXXXXXXX from disk cache
[09:38:22] [trace] SdkProvider#forEnvironment()
[09:38:22] [trace]   SdkProvider#resolveEnvironment()
[09:38:22] [trace]   SdkProvider#obtainBaseCredentials()
[09:38:22] [trace]     SdkProvider#defaultAccount()
[09:38:22] [trace]     SdkProvider#defaultCredentials()
[09:38:22] [trace]   SdkProvider#withAssumedRole()
[09:38:22] Assuming role 'arn:aws:iam::XXXXXXXXX:role/cdk-abc1234-deploy-role-XXXXXXXXXX-us-west-2'.
[09:38:22] [trace]   SDK#forceCredentialRetrieval()
[09:38:22] [trace] SDK#cloudFormation()
[09:38:22] [trace]   SDK#wrapServiceErrorHandling()
[09:38:22] Waiting for stack CDKToolkit to finish creating or updating...
[09:38:22] [AWS cloudformation 400 0.373s 0 retries] describeStacks({ StackName: 'CDKToolkit' })
[09:38:22] [trace] SDK#makeDetailedException()
[09:38:22] Call failed: describeStacks({"StackName":"CDKToolkit"}) => Stack with id CDKToolkit does not exist (code=ValidationError)
[09:38:22] Stack CDKToolkit does not exist
[09:38:22] The environment aws://XXXXXXXXXXX/us-west-2 doesn't have the CDK toolkit stack (CDKToolkit) installed. Use cdk bootstrap "aws://XXXXXXXXXXXX/us-west-2" to setup your environment for use with the toolkit.
The template for stack "CdkLargeTemplateStack" is 78KiB. Templates larger than 50KiB must be uploaded to S3.
Run the following command in order to setup an S3 bucket in this environment, and then re-deploy:

        $ cdk bootstrap aws://XXXXXXXXXXXX/us-west-2

[09:38:22] Template too large to deploy ("cdk bootstrap" is required)

On the line with the "describeStacks" call, the diff action is trying to describe the "CDKTookit" stack, however when looking at the diff output of a template less that 50KiB that "describeStacks" call would contain the name of the stack that you are running the diff against and not the "CDKToolkit" stack.

[09:52:10] Assuming role 'arn:aws:iam::XXXXXXXXXXXX:role/cdk-abc1234-deploy-role-XXXXXXXXX-us-west-2'.
[09:52:10] [trace]   SDK#forceCredentialRetrieval()
[09:52:10] [trace] SDK#cloudFormation()
[09:52:10] [trace]   SDK#wrapServiceErrorHandling()
[09:52:11] [AWS cloudformation 200 0.395s 0 retries] describeStacks({ StackName: 'CdkLargeTemplateStack' })
Hold on while we create a read-only change set to get a diff with accurate replacement information (use --no-change-set to use a less accurate but faster template-only diff)
[09:52:11] Removing existing change set with name cdk-diff-change-set if it exists
[09:52:11] [AWS cloudformation 200 0.397s 0 retries] deleteChangeSet({
  StackName: 'CdkLargeTemplateStack',
  ChangeSetName: 'cdk-diff-change-set'
})

The situation remains the same even if your account does have a bootstrap stack named "CDKToolkit". In that case the diff action uploads the synthesized template to the asset bucket of the "CDKToolkit" stack, but you then end up with an error like this since the custom stack execution role is trying to access a template in the "CDKToolkit" asset bucket.

[09:33:04] Call failed: createChangeSet({"StackName":"CdkLargeTemplateStack","ChangeSetName":"cdk-diff-change-set","ChangeSetType":"UPDATE","Description":"CDK Changeset for diff 75f8fef6-ad79-48ad-b15e-561ec904d608","ClientToken":"diff75f8fef6-ad79-48ad-b15e-561ec904d608","TemplateURL":"https://cdk-hnb659fds-assets-XXXXXXXXXX-us-west-2.s3.us-west-2.amazonaws.com/cdk/CdkLargeTemplateStack/bb39bade6a75e210f96c66386e09567d088d1d72d9b75007d4b04084b62b569c.yml","Parameters":[],"RoleARN":"arn:aws:iam::XXXXXXXXXX:role/cdk-abc1234-cfn-exec-role-XXXXXXXXXX-us-west-2","Capabilities":["CAPABILITY_IAM","CAPABILITY_NAMED_IAM","CAPABILITY_AUTO_EXPAND"]}) => S3 error: Access Denied
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html (code=ValidationError)
[09:33:04] S3 error: Access Denied
For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
Could not create a change set, will base the diff on template differences (run again with -v to see the reason)

Reproduction Steps

1. Bootstrap your account with:

   cdk bootstrap aws://<Account>/us-west-2 --toolkit-stack-name "custom-stack" --qualifier "abc1234"
   

2. Add "@aws-cdk/core:bootstrapQualifier": "abc1234" to your cdk.json

3. Remove the default "CDKToolkit" stack if it exists

4. Perform a "cdk diff -vvv" of an application with a synthesized template larger than 50KiB

Possible Solution

No response

Additional Information/Context

This isn't a high priority issue since in call cases the "cdk deploy" works correctly. You just don't get a proper diff

Additionally, if you add "toolkitStackName" to your cdk.json or specify "--toolkit-stack-name <Stackname>" to your "cdk diff" command, then the diff works as exected.

The issue seems to have been introduced in aws-cdk 2.119.0 and is not present in 2.118.0

CDK CLI Version

2.128.0

Framework Version

No response

Node.js Version

18.18

OS

MacOS 14.3.1

Language

TypeScript

Language Version

5.3.3

Other information

No response

[pahud]

related to https://github.com/aws/aws-cdk/issues/28249

[diranged]

Seeing this as well - luckily the --toolkit-stack-name hack helped us get around it for now.