arcade-services
arcade-services copied to clipboard
dotnet
Improvements to DarcLib's GitHub app usage to reduce API load
Migrated from https://github.com/dotnet/core-eng/issues/13732
@MattGal wrote:
Follow up from https://github.com/dotnet/core-eng/issues/13695. At least when filing this isn't really FR as we've been granted a substantial increase in max API usage in addition to finding the primary source for such usage and quieting it down, so I'm filing this to discuss in triage.
There are three improvements we can make to the way we use the dotnet-maestro github app to prevent exceeding API quota, since even with nothing going wrong we do slowly grow the number of repos and subscriptions over time.
- ~Set a maximum number of tries for Dependency Flow attempts. As seen in https://github.com/dotnet/core-eng/issues/13703, it may keep trying to trigger a subscription for a dead branch forever (in this case, ever 5 minutes for almost a week). This should be an error scenario, but it should maybe give up after 10, or 20, or 100 attempts. Any number that is less than infinity.~
- ExecuteRemoteGitCommandAsync needs to at least detect HTTP 403s and check for rate-limiting behavior (weird as this is, it's extremely unlikely to change). The logic is pretty simple, and it can be taught to either return a specific retry-able error, or simply retry by itself after the wait time.. All you need to know is in https://docs.github.com/en/rest/overview/resources-in-the-rest-api#rate-limiting. Alternatively, this same API could check dotnet-maestro's rate limiting status BEFORE the first attempt, as this call does not count against quota.
- DarcLib's GitHub API usage could use conditional requests when stuck in polling loops, to get HTTP 304s (the "nothing changed" response), which also don't get counted against API quota. Doing this just for the polling behavior of pull request actors would likely provide a big improvement.
@riarenas , @epananth FYI.
- We now have 3 attempts.
- We tried this in https://github.com/dotnet/arcade-services/pull/4421 but it would not actually help us that much, if we paused everything at once and then slammed GitHub later from all the threads.
- This is a great idea that we should try.
