CDK deploy: Cloudfront Distribution resources update fail with rate exceeded errors

[shaelm]

Describe the bug

Deploying multiple stacks each with their own cloudfront distribution, causes a deployment failure

in have tried in both 2.127.0, and 2.92.0 with the same issue:

Rate exceeded for operation 'AWS::CloudFront::Distribution'.

when trying to update all my nested stacks (around 130) each with their own cloudfront distribution.

Expected Behavior

There should be a retry mechanism with backoff, instead of failing the deployment. There is currently no way to control that in the cdk deployment, so my deployments fail. I know we can add dependencies between the nested stacks, but that seems like not a great solution to add a stack dependency for the sake of deployment throttling to update API's

There was a similar issue with log retention: https://github.com/aws/aws-cdk/issues/26837

Current Behavior

Resource update fails with:

Rate exceeded for operation 'AWS::CloudFront::Distribution'.

Deployment is cancelled and rolled back.

Reproduction Steps

Deploy an update to > 100 cloudfront distributions in a cdk deployment

Possible Solution

CDK deployment should retry and do a backoff if getting throttled by cloudfront update API

Additional Information/Context

No response

CDK CLI Version

2.127

Framework Version

No response

Node.js Version

18

OS

MAc OS 14.2.1

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

CDK does not handle the deployment failure due to rate limit exceeded and we need more discussion about that. At this moment, I guess you will need to ensure the distributions dependency using addDependency() and make sure multiple distributions would not be deployed at the same time.

[shaelm]

Is there no other workaround?

I'm hesitant to add dependencies between stacks for the sake of it that really aren't dependent on each other just to slow down the deployment

[pahud]

Unfortunately this is probably the only workaround from CDK as the limitation is from CFN and service, not CDK.

Another approach off the top of my head is to reach out to the AWS support to discuss your use case and see if it's possible to raise the limit of your AWS account but that is beyond the scope of CDK.

[shaelm]

In https://github.com/aws/aws-cdk/issues/26837, they implemented a cdk property, logRetentionRetryOptions?, (source: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_logs.LogRetention.html_) which solved this exact problem due to a low tps API from the service team.

Could we do something similar for cloudfront distribution constructs?

[pahud]

logRetentionRetryOptions is for CDK custom resource which CDK has the control over it but deploying 130+ cloudfront distributions at the same time could hit the service limit from the cloudffront team. This could happen even when you just use cloudformation without CDK.

This is not something CDK can do to work it around except ensuring the dependency. I suggest you create a premium support ticket and discuss this with the premium support team to escalate your concern to the service team.

[github-actions[bot]]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.