group-sync-operator icon indicating copy to clipboard operation
group-sync-operator copied to clipboard

Keep requests to OKTA under rate limits

Open dweebo opened this issue 3 years ago • 0 comments

We are seeing an issue where because we are syncing a large number of groups from OKTA the group-sync-operator makes a large number of requests to OKTA and we are exceeding our rate limit.

For example we have a rate limit of 500 requests / minute. When syncing to a cluster with 250 groups we exceed that rate limit, because this operator makes 2x calls for each group (to get group details and to get group membership).

I think because this project uses the OKTA go client it automatically retries when rate limited so groups are eventually synced correctly. However during the time we are over the rate limit all of our OTHER uses of the OKTA API are affected and it is affecting our business.

I'm not sure the best solution but I'd like to consider modifying this operator so that it looks at the rate limit headers returned from requests and slows down it's requests when it nears the rate limit to stay under the limit. https://developer.okta.com/docs/reference/rl-best-practices/

dweebo avatar Feb 24 '22 21:02 dweebo