stacklok
Handle rate limiting errors for GitHub API
Describe the issue
Currently, we don't handle rate limit exceeded errors from the GitHub API. From GitHub API Docs:
Continuing to make requests while you are rate limited may result in the banning of your integration.
GitHub suggests Best practices for using the REST API, which mentions handling the rate limit exceeded errors.
Discord Thread: https://discord.com/channels/1184987096302239844/1185287949240242258/1201083912521252885
- [x] Add client caching to handle rate limiting (https://github.com/stacklok/minder/pull/2271)
- [ ] Update all endpoints to handle rate-limiting errors
To Reproduce
No response
What version are you using?
main
Rate-limiting errors are on a token/user basis. Blocking the goroutine by storing some state about the token should prevent Minder from making repeated requests using that token, preventing the integration from getting banned.
Currently, Minder uses gochannel and SQL pub-sub implementation from Watermill to handle incoming events. Both are single channels with head-of-line blocking, which means that blocking in middleware/handler blocks all processing. This means blocking any goroutine (created by event handlers/middleware) using current pub-sub implementations would block the complete event processing.
However, given the current scale, getting rate limited is rare, and over the last month we've observed zero 403 or 429 responses (See this comment on discord), so having blocking logic until we replace with some other pub-sub implementation shouldn't be a problem.
Add metrics and an alert to determine if this is still happening. If it is, we will deal with it.
