[Bug]: Organizations API throttling
Is there an existing issue for this?
- [x] I have searched the existing issues
Describe the bug
In an installation with a fair amount of accounts (more than 250) and a pretty Organization structure, we're facing multiple issues regarding Organizations API throttling.
Most of the times, the throttling regard ListParents and ListRoots API calls
These throttlings cause slowdowns or errors in the execution of the step functions adf-account-management and adf-account-bootstrapping, especially in the node EnableBootstrappingJumpRole.
These are the output we observed on the EnableBootstrappingJumpRole node:
{
"error": "Task failed. Granting the ADF Account-Bootstrapping Jump Role privileged cross-account access failed due to an error: An error occurred (TooManyRequestsException) when calling the ListParents operation (reached max retries: 4): You have sent too many requests in too short a period of time. Try again later.."
}
{
"error": "Task failed. Granting the ADF Account-Bootstrapping Jump Role privileged cross-account access failed due to an error: An error occurred (TooManyRequestsException) when calling the ListRoots operation (reached max retries: 4): You have sent too many requests in too short a period of time. Try again later.."
}
Expected Behavior
The step functions should complete without errors, and perform in a timely manner
Current Behavior
Step functions randomly fail, take very long time to complete (~50 minutes)
Steps To Reproduce
No response
Possible Solution
Use some form of caching for Organizations API calls
Additional Information/Context
No response
ADF Version
4.0.0
Contributing a fix?
- [x] Yes, I am working on a fix to resolve this issue