one-observability-demo icon indicating copy to clipboard operation
one-observability-demo copied to clipboard

CDK deploy is failing with TooManyRequestsException

Open amayaea opened this issue 2 years ago • 1 comments

Running cdk deploy --context admin_role=$EKS_ADMIN_ARN Services --context dashboard_role_arn=$CONSOLE_ROLE_ARN --require-approval never fails with message: Services: creating CloudFormation changeset... 6:36:00 PM | CREATE_FAILED | Custom::AWSCDK-EKS-KubernetesResource | loadBalancerServiceAccount1D19AD3A Received response status [FAILED] from custom resource. Message returned: TooManyRequestsException: Rate Exceeded. at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27) at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8) at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20) at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10) at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14) at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10) at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12) at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10 at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9) at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12) (RequestId: 36cd730b-cc24-4222-a36a-ff0c219804bf) Screen Shot 2022-08-03 at 4 03 52 PM

amayaea avatar Aug 03 '22 20:08 amayaea

Seeing the same behavior - most probably caused by https://github.com/aws-samples/one-observability-demo/pull/176 as it appears Lambda is getting throttled.

deekob avatar Aug 10 '22 04:08 deekob

Folks, this is an AWS internal situation where Isengard throttles resources (low containment score). Try again in an older Isengard account and you should be good. You could also test this on Event Engine. Closing this thread.

awsimaya avatar Oct 24 '22 15:10 awsimaya