fix: use retry and throttle octokit plugins

[achingbrain]

Refactors the client to use @octokit/plugin-retry and @octokit/plugin-throttling as GitHub occasionally changes it's API and these plugins can abstract those changes away from this module.

• Removes lib/definitions/rate-limit.js
• Adds lib/definitions/retry.js and lib/definitions/throttle.js to handle retry/throttle settings
• Updates tests to be more like GitHub (returing the correct rate limit response headers, etc)

Fixes #299 and https://github.com/semantic-release/semantic-release/issues/2204 See also #480 and #378

[achingbrain]

I'm trialing this fix in a bunch of repos - so far, so good.

If anyone else wants to give it a go you can use my temporary fork with this patch applied.

Change your @semantic-release/github dep in package.json to be:

"@semantic-release/github": "https://registry.npmjs.org/@achingbrain/semantic-release-github/-/semantic-release-github-0.0.0.tgz",

No other changes required.

[lapastillaroja]

It would be great if this PR can be reviewed and marged. I'm facing same problem in one of my projects

[rathpc]

Would love to get this reviewed and merged!! Can we update to resolve conflicts and tag some people for review?

[achingbrain]

Conflicts are gone, not sure who would review - @travi @gr2m ?

[gr2m]

I'm still on parental leave, and first week back to work will be busy, but I have it on my radar

[PocketlawErik]

@gr2m Wonderful to hear!

Running into this same problem right now. All your hard work is much appreciated! Thanks!

Please let me know if there is anything I can do to help 🙏❤️

[xorph]

@gr2m Any ETA for a review of this PR? We have to deal a lot with the secondary rate limits in our release process lately.

[gr2m]

No update yet, sorry. I'm still on parental leave and life keeps me busy. We focus the little time we have right now on #418

[stipsan]

Hi @gr2m, our usage of semantic-release at sanity.io is only increasing as it's such a boost to our productivity.

We're also impacted by this problem now and then, is there a way we can help @gr2m? Either by reviewing and fixing things adjacent to ESM or would you prefer us to join the effort to land ESM #418? We've been doing a lot of ESM migrations in our own projects this year so it's familiar grounds to us 🥲

[achingbrain]

@travi @gr2m is there any chance you could find some time to review this PR? I've updated it to resolve the recent conflicts, it solves a very real problem when using this module in monorepos that publish lots of modules in one go.

Please let me know if there's anything I can do to help you get it across the line?

[rlch]

+1 -- we constantly run into rate-limiting issues in our delivery workflows. Getting quite tedious having to constantly monitor and manually retry failed actions.

[rathpc]

sorry for pinging directly but we really need traction on this PR to get it merged. It has been 8 months already and this issue is plaguing many people.

Can one of you review and get this merged please? @boennemann @gr2m @kbrandwijk @keplersj @pvdlg @travi @UziTech

[Aeolun]

If nobody currently working on this project has time to review/merge these changes, is there a chance someone else can get review/merge permissions? 8 months for a PR that looks fairly straightforward (and presumably passes tests) is a bit much.

Forking is always an option, but less than ideal because everyone will have to make their own fixes (or run into the problem, find this thread, and then pull the fixed version that someone already published to npm).

[rathpc]

another 2 weeks bites the dust and another 30 or so failures I have experienced due to this secondary rate limit issue.

What can we do to get this merged and released please?

[tractorcow]

Thanks @achingbrain I'm using this in my github action and it seems to work well (will report if I run into any rate limiting issues).

npm install -g \
          @semantic-release/github@npm:@achingbrain/[email protected] \
          semantic-release \
          @semantic-release/exec \
          @semantic-release/git

[tractorcow]

Looks like I spoke too soon. I have the issue even with the suggested fork. However, even with retry and delay rate limit errors are still possible.

[1:55:15 AM] [semantic-release] › ℹ  Start step "publish" of plugin "@semantic-release/github"
(node:1886) ExperimentalWarning: The Fetch API is an experimental feature. This feature could change at any time
(Use `node --trace-warnings ...` to show where the warning was created)
[1:55:16 AM] [semantic-release] [@semantic-release/github] › ℹ  Published GitHub release: https://github.com/pixelfusion/mindlab-wl-academyex-website/releases/tag/v1.20.0-uat.39
[1:55:16 AM] [semantic-release] › ✔  Completed step "publish" of plugin "@semantic-release/github"
[1:55:16 AM] [semantic-release] › ℹ  Start step "publish" of plugin "@semantic-release/exec"
[1:55:16 AM] [semantic-release] › ✔  Completed step "publish" of plugin "@semantic-release/exec"
[1:55:16 AM] [semantic-release] › ℹ  Start step "success" of plugin "@semantic-release/github"
[1:55:17 AM] [semantic-release] › ✘  Failed step "success" of plugin "@semantic-release/github"
[1:55:17 AM] [semantic-release] › ℹ  Start step "success" of plugin "@semantic-release/exec"
[1:55:17 AM] [semantic-release] [@semantic-release/exec] › ℹ  Call script echo 'v1.20.0-uat.39' > version.txt
[1:55:17 AM] [semantic-release] › ✔  Completed step "success" of plugin "@semantic-release/exec"
[1:55:17 AM] [semantic-release] › ✘  An error occurred while running semantic-release: RequestError [HttpError]: You have exceeded a secondary rate limit. Please wait a few minutes before you try again.
    at /opt/hostedtoolcache/node/18.12.1/x64/lib/node_modules/semantic-release/node_modules/@octokit/request/dist-node/index.js:88:21

[tractorcow]

@achingbrain is there an issue with the way I'm deploying your fix, or is the fix legitimately not catching the rate limiting?

[carlobernardini]

Respectfully, is this project still maintained? This pr has been open for almost a year, the last merge is of a similar date. It would be really great if we can get this merged since the issue is hurting many.

[achingbrain]

@tractorcow you can see how I'm using the plugin here.

Unfortunately there are still some bugs in the throttle plugin itself, though those will be fixed there and not here.

[tractorcow]

Thanks for getting back to me.

For now i'm using semantic-release || true in my release scripts, because even if github fails, my code still gets deployed in that case anyway :)

[alexsanderp]

News?

[RyKilleen]

For what it's worth, this challenge is the one blocker for adoption, both for me and the organizations I work with.

I think semantic release is such a great tool, being able to use it without the friction of half failed releases in the GH Actions would be awesome!

[JeanGautierSherpas]

Hello, Big fan of semantic-release, I used it every where in project i worked for. This is issue is causing trouble for my current organisation, is it possible to accelerate solving ? Need any hand ?

[gr2m]

I will prioritize this issue once we ship the ESM updates. I'll plan on working on this after April 14th

[rebeljoi]

For those of you who still experience this problem. I solved this in the meantime by disabling the features in the @semantic-release/github plugin that uses the https://api.github.com/search endpoint, which is the one that causes the secondary rate limit exceeds.

Configure @semantic-release/github as follows:

[
  "@semantic-release/github",
  {
    "successComment": false,
    "failTitle": false
  }
]

[achingbrain]

Godspeed @gr2m and massive ❤️ for picking this up again, it's a vital part of so many people's workflows.

I've noticed a lot of people are linking to this PR saying it will solve the rate limiting problems - they should know it's only part of the solution, https://github.com/octokit/plugin-throttling.js/issues/454 needs addressing too.

[gr2m]

Okay more like May 12 but that's how life goes sometimes. Thank you all for your patience and your help. I'm looking into it now

[gr2m]

It's happening dot gif

If I had more time I'd add tests for lib/semantic-release-octokit.js but it's a tad complicated if we make it right, and tests will change in the upcoming ESM version, so chose to ... not.

I'm looking into the failing Node 14.17 tests. If y'all could give the latest version a try, I'd be much obliged

[gr2m]

I published the current version as @gr2m/[email protected] for easier testing.

You can use aliasing to test it out. In your package.json

  "devDependencies": {
    "semantic-release": "^21.0.2",
    "@semantic-release/github": "npm:@gr2m/[email protected]"
  },

[gr2m]

Alright I tested it some, it's Saturday, let's give this a go. I'm confident in this not introducing any new problems, but we can really only see once it's used out there.