You have exceeded a secondary rate limit

[timja]

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

Run multi-gitter with this config: https://github.com/hmcts/multi-gitter-backstage/tree/b52250d5554d1f4b903d81e1dfc667137e9b4eff

After about 50 repositories I started getting this quite frequently:

INFO[0395] POST https://api.github.com/repos/hmcts/fpl-ccd-configuration/pulls: 403 You have exceeded a secondary rate limit and have been temporarily blocked from content creation. Please retry your request again later. []

Expected behavior

Create the pull requests slow enough that you don't get rate limited (and also sleep and retry if it does happen)

Additional context Was supposed to be fixed by https://github.com/lindell/multi-gitter/pull/207

[lindell]

Hello @timja and thanks for the bug report.

It would indeed be good if multi-gitter could be aware of the rate limit. The previous fix you referred to (#207) was to fix the limit of 1 creation per second and was not intended to fix the overall limit.

The limit seems to be 5000 requests for free user tokens and reset every hour. Multi-gitter uses one request per repository + one extra if a pull request is created (+1 request per 100 repositories in the initial fetch). So with a fresh token, you should be able to run on 2475 repositories assuming all will create a pull request. The reset time does also seem to be once an hour. So it is likely that if the limit is reached, multi-gitter has to wait around 30 minutes or more before continuing. So I think in general, you should try to keep §

Specific to your use case: It seems that the @hmcts org has 1200 (public) repositories. Assuming you run with a token that has not yet depleted most of the rate limit, you should be fine. Could you please confirm if the token has been used for anything else, or in a previous multi-gitter run when this occurred.

[timja]

The token should be barely used, I was hitting secondary rate limiting at a far lower limit than that, creating the pull request was the issue regular rate limit was fine

[lindell]

You are right, this is another example of the secondary limit. The documentation is unclear on how often you can make request / should wait for requests if they are rejected. It is said that the Retry-after header should be present, but aren't always present when requests such as created pull requests are made. I think a combination of looking at that headers, and a exponential backoff needs to be made.

https://docs.github.com/en/rest/guides/best-practices-for-integrators#dealing-with-secondary-rate-limits

[lindell]

A PR for this retry with backoff has been creates #280

Could you install and try it with:

go install github.com/lindell/multi-gitter@retry-tmp

[timja]

Thanks! I'll try give it a go.

[lindell]

Have you had the time to test this out yet @timja?

[timja]

I haven't yet, I do have a use-case for this coming up. but I'm going on leave for a week, so it may be a week or two before I get back to this.

If your testing shows this is an improvement I would suggest merging this, and I'll report my findings back when I do get to run this.

[github-actions[bot]]

Included in release v0.42.4 🎉