automated-security-response-on-aws

automated-security-response-on-aws copied to clipboard

Hi, I was trying to install AWS SHARR templates in one of our accounts. It is failing due to a rate limitation on the Systems manager. How do we prevent this?

Root Cause Error in text format: Received response status [FAILED] from custom resource. Message returned: An exception occurred: ClientError: An error occurred (ThrottlingException) when calling the DeleteDocument operation (reached max retries: 9): Rate exceeded

Ref to AWS SHARR clouformation templates. [https://docs.aws.amazon.com/solutions/latest/automated-security-response-on-aws/templates.html

Thanks for letting us know about this. We have seen this happen at least one other time, so our custom resource provider for SSM documents is not fully able to rate-limit itself. We will explore ways to fix this in a future release.

One option would be to eliminate our custom resource provider. Enhancements to the CFN behavior with SSM documents may have made this possible.

Otherwise, we can improve the rate-limiting in the provider to make it more robust. Using Lambda reserved concurrency to limit the provider to serial invocations means that each invocation is running up the adaptive backoff retry count in the SSM client until one of them fails as in your case. In this use case a fixed backoff or at least a larger starting delay would be more appropriate.

This should be fixed in v1.5.1, see #74