Deal with GitHub's secondary rate limit

[fthomas]

My public Scala Steward instance has repeatedly hit GitHub's secondary rate limit recently when creating PRs:

org.scalasteward.core.util.UnexpectedResponse: uri: https://api.github.com/repos/***/***/pulls
method: POST
status: 403 Forbidden
headers: Headers(
  access-control-allow-origin: *,
  access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset
  content-security-policy: default-src 'none'
  content-type: application/json; charset=utf-8
  date: Wed, 10 Nov 2021 04:26:48 GMT
  referrer-policy: origin-when-cross-origin
  strict-origin-when-cross-origin
  server: GitHub.com
  strict-transport-security: max-age=31536000; includeSubdomains; preload
  vary: Accept-Encoding, Accept, X-Requested-With
  x-accepted-oauth-scopes: 
  x-content-type-options: nosniff
  x-frame-options: deny
  x-github-media-type: github.v3
  x-github-request-id: B1C8:AD73:2D8A937:2E660AE:618B4A03
  x-oauth-scopes: delete_repo
  notifications, repo, workflow
  x-ratelimit-limit: 5000
  x-ratelimit-remaining: 4290
  x-ratelimit-reset: 1636520514
  x-ratelimit-resource: core
  x-ratelimit-used: 710
  x-xss-protection: 0
  )
body: {
  "message":"You have exceeded a secondary rate limit and have been temporarily blocked from content creation. Please retry your request again later.",
  "documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#secondary-rate-limits"
}

GitHub docs about the secondary rate limit are here:

• https://docs.github.com/en/rest/overview/resources-in-the-rest-api#secondary-rate-limits
• https://docs.github.com/en/rest/guides/best-practices-for-integrators#dealing-with-secondary-rate-limits

The response doesn't include a Retry-After header, so we don't know how long Scala Steward should wait before creating the next PR.

[d-g-n]

Hello, we're trying to run a local version of scala-steward against approx 30-35 repos, but given how out of date they are, we're frequently getting body: {"message":"You have exceeded a secondary rate limit and have been temporarily blocked from content creation. Please retry your request again later.","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#secondary-rate-limits"},

i acknowledge that github isn't returning the correct header but is there any config or setting we can tweak to hopefully mitigate this until a full fix is in place?

[htmldoug]

Looks like github returns Retry-After now!

When you have been limited, use the Retry-After response header to slow down. The value of the Retry-After header will always be an integer, representing the number of seconds you should wait before making requests again. For example, Retry-After: 30 means you should wait 30 seconds before sending more requests.

https://docs.github.com/en/rest/guides/best-practices-for-integrators#dealing-with-secondary-rate-limits

Actual response I received:

org.scalasteward.core.util.UnexpectedResponse: uri: https://api.github.com/repos/***/***/pulls?head=****/***%3Aupdate/google-api-services-sheets-v4-rev20220221-1.32.1&base=main&state=all
method: GET
status: 403 Forbidden
headers:
  access-control-allow-origin: *
  access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
  content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com objects-origin.githubusercontent.com www.githubstatus.com collector.githubapp.com collector.github.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events translator.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src render.githubusercontent.com viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com collector.github.com avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com objects-origin.githubusercontent.com secured-user-images.githubusercontent.com/; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; worker-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/
  content-type: application/json; charset=utf-8
  date: Thu, 03 Mar 2022 16:14:51 GMT
  expect-ct: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
  referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
  retry-after: 60
  server: GitHub.com
  strict-transport-security: max-age=31536000; includeSubdomains; preload
  vary: Accept-Encoding, Accept, X-Requested-With
  x-content-type-options: nosniff
  x-frame-options: deny
  x-github-media-type: github.v3; format=json
  x-github-request-id: AEC9:5D27:43638DF:7FCD679:6220E961
  x-xss-protection: 0
body: {
  "documentation_url": "https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits",
  "message": "You have exceeded a secondary rate limit. Please wait a few minutes before you try again."
}

[fthomas]

You got this header in response to a GET for listing PRs. I posted a response to a POST for creating a PR. The documentation you cited also mentions that there is no RetryAfter for creating PRs. So I doubt that GitHub has changed anything here.

[htmldoug]

Ah, good point. So #2540 would only fix a subset of these issues. The majority of the failures I'm seeing are actually from POST .../forks, so I can at least take care of those. I'll update my PR so it keeps this issue open.

org.scalasteward.core.util.UnexpectedResponse: uri: https://api.github.com/repos/***/***/forks
method: POST
status: 403 Forbidden
headers:
  access-control-allow-origin: *
  access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
  content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com objects-origin.githubusercontent.com www.githubstatus.com collector.githubapp.com collector.github.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com cdn.optimizely.com logx.optimizely.com/v1/events translator.github.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src render.githubusercontent.com viewscreen.githubusercontent.com notebooks.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com collector.github.com avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com objects-origin.githubusercontent.com secured-user-images.githubusercontent.com/; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; worker-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/
  content-type: application/json; charset=utf-8
  date: Thu, 03 Mar 2022 16:14:51 GMT
  expect-ct: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
  referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
  retry-after: 60
  server: GitHub.com
  strict-transport-security: max-age=31536000; includeSubdomains; preload
  vary: Accept-Encoding, Accept, X-Requested-With
  x-content-type-options: nosniff
  x-frame-options: deny
  x-github-media-type: github.v3; format=json
  x-github-request-id: AEC9:5D27:4363908:7FCD6C5:6220E97B
  x-xss-protection: 0
body: {
  "documentation_url": "https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits",
  "message": "You have exceeded a secondary rate limit. Please wait a few minutes before you try again."
}

[exoego]

Regarding #2540, code LGTM 👍 However, secondary rate limit for POST does not return Retry-After header, so I am afraid that adding Retry-After is going to be an overhead.

From the document,

If you're making a large number of POST, PATCH, PUT, or DELETE requests for a single user or client ID, wait at least one second between each request.

I wonder adding sleep(1.second) probably mitigate secondary rate limit.

[htmldoug]

Thanks for the review of #2540!

The document also says:

When you have been limited, use the Retry-After response header to slow down. The value of the Retry-After header will always be an integer, representing the number of seconds you should wait before making requests again. For example, Retry-After: 30 means you should wait 30 seconds before sending more requests.

Implementing both of the github best practices seems reasonable to me--probably best as separate PRs.

I am afraid that adding Retry-After is going to be an overhead.

It's true that not all requests would return Retry-After. I haven't benchmarked it, although I expect that checking for the presence of the header should have negligible CPU/memory cost. Additionally, I'd be surprised if this code path were hot enough to optimize.

Did you mean this as a blocker for #2540? Is there anything else I can do to address concerns so that this can be merged?

We have a long list of repos we run scala-steward against and it's currently failing for us at the end of that list. I'd love to see a fix merged one way or another.

[exoego]

I've merged https://github.com/scala-steward-org/scala-steward-action/issues/335 It does not cover all cases but helps some cases definitely.

I keep this issue open until we have solutions for other cases.

[rossabaker]

We are still struggling with this. I wonder if the Retry-After from the other endpoints gives us some empirical guess as to what would be an appropriate time to sleep? Like, is it always a fixed duration, or is it (more likely) some token bucket and varies wildly from run to run?

[rossabaker]

Or if we know whether we're talking on the order of resting for a couple seconds vs "a few minutes" like the message. A couple seconds seems worthwhile. "A few minutes" could really add up on runners where we pay by the minute.

[fthomas]

I don't maintain an instance that works with GitHub anymore, but still have logs of @scala-steward from March and April 2022. I looked at four occasions where it hit the secondary rate limit and it was able to create PRs again after ~ 40, 30, 8, and 40 minutes.

[fthomas]

I don't know if waiting for a few seconds or minutes between PRs would have meant that it would have never hit the secondary rate limit.