aws-deployment-framework icon indicating copy to clipboard operation
aws-deployment-framework copied to clipboard

[Bug]: Organizations API throttling

Open igordust opened this issue 1 year ago • 0 comments

Is there an existing issue for this?

  • [x] I have searched the existing issues

Describe the bug

In an installation with a fair amount of accounts (more than 250) and a pretty Organization structure, we're facing multiple issues regarding Organizations API throttling.

Most of the times, the throttling regard ListParents and ListRoots API calls

These throttlings cause slowdowns or errors in the execution of the step functions adf-account-management and adf-account-bootstrapping, especially in the node EnableBootstrappingJumpRole.

These are the output we observed on the EnableBootstrappingJumpRole node:

{
  "error": "Task failed. Granting the ADF Account-Bootstrapping Jump Role privileged cross-account access failed due to an error: An error occurred (TooManyRequestsException) when calling the ListParents operation (reached max retries: 4): You have sent too many requests in too short a period of time. Try again later.."
}
{
  "error": "Task failed. Granting the ADF Account-Bootstrapping Jump Role privileged cross-account access failed due to an error: An error occurred (TooManyRequestsException) when calling the ListRoots operation (reached max retries: 4): You have sent too many requests in too short a period of time. Try again later.."
}

Expected Behavior

The step functions should complete without errors, and perform in a timely manner

Current Behavior

Step functions randomly fail, take very long time to complete (~50 minutes)

Steps To Reproduce

No response

Possible Solution

Use some form of caching for Organizations API calls

Additional Information/Context

No response

ADF Version

4.0.0

Contributing a fix?

  • [x] Yes, I am working on a fix to resolve this issue

igordust avatar Feb 26 '25 14:02 igordust