HTTPError: Response code 429 (Too Many Requests)

[devinshoemaker]

I believe that dev.to has a load balancer or something that prevents too many requests from a single host from being fired off too quickly. Both locally and in my CI/CD pipeline, I frequently get these errors. Due to these errors, it sometimes takes executing dev-to-git multiple times to get my posts updated. I think that a slight delay added between publishing posts could avoid this issue.

[DRAFT] Article "CI Test Success" encountered an error:
Error name: "HTTPError"
Error message: "Response code 429 (Too Many Requests)"
Error stack: "HTTPError: Response code 429 (Too Many Requests)
    at EventEmitter.<anonymous> (/Users/devin/code/nx/playground/nxtend-playground/node_modules/dev-to-git/bin/dev-to-git.umd.js:7080:23)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)"
[PUBLISHED] Article "How to Configure Prettier to Automate Code Formatting" is already up to date
[PUBLISHED] Article "Developing Ionic React Apps in an Nx Workspace with @nxtend/ionic-react" is already up to date
[PUBLISHED] Article "Release @nxtend/ionic-react 2.0.0" is already up to date
[PUBLISHED] Article "@nxtend/ionic-react 2.2.0" encountered an error:
Error name: "HTTPError"
Error message: "Response code 429 (Too Many Requests)"
Error stack: "HTTPError: Response code 429 (Too Many Requests)
    at EventEmitter.<anonymous> (/Users/devin/code/nx/playground/nxtend-playground/node_modules/dev-to-git/bin/dev-to-git.umd.js:7080:23)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)"
[PUBLISHED] Article "Build Cross-Platform Applications in a Monorepo with Nx, Ionic, and Capacitor" is already up to date
[PUBLISHED] Article "Building an Nx Plugin to Add Dependencies to a Project" is already up to date
[PUBLISHED] Article "Creating an ng add Schematic for an Nx Plugin" is already up to date
[PUBLISHED] Article "Executing External Schematics in an Nx Plugin" is already up to date
[PUBLISHED] Article "Extending Schematics in an Nx Plugin" is already up to date
[PUBLISHED] Article "Running Nx Affected Commands in GitHub Actions" is already up to date
[PUBLISHED] Article "Nx Brings High-Quality CLI Tooling to React" is already up to date

[devinshoemaker]

@beeman has also experienced this and may be able to provide more information.

[maxime1992]

There's indeed a "Too Many Requests" error code exposed in the API: https://docs.dev.to/api/#operation/updateArticle

I couldn't find the delay though :thinking:

If you're willing to work on a fix I can assign this issue to you :smile_cat: Maybe we can set an arbitrary delay of 500ms in a constant for now (unless someone finds the actual exact time :pray:)

[zakhenry]

It's probably fastest to let the too many requests happen, then retry after the retry-after header says so: https://github.com/thepracticaldev/dev.to/blob/5b62811c9874fa2770fe3f720abac1f80fe6279f/spec/requests/api/v0/articles_spec.rb#L564

[maxime1992]

Indeed, sounds like a good plan :+1:

[devinshoemaker]

@maxime1992 do you still want me to try and tackle this then? It should be easy to check for the retry-after response header. I should have a little bit of time this evening.

[maxime1992]

@devinshoemaker if you're keen to, sure :)

[devinshoemaker]

I'll give it a go! It'd be nice to have this ready for the Nx plugin release.

[devinshoemaker]

I'm a bit stumped, to be honest. I'm not familiar with the got library so I've been doing a little bit of research, and it seems the docs should be able to handle the retry-after header gracefully.

Docs: https://github.com/sindresorhus/got#retry

I found the values for the rate limit on publishing posts. I'm not sure if the got maxRetryAfter property accepts seconds or milliseconds, so I tried both to be sure. However, with this change, I saw no improvements to the HTTP 429 error.

Here is my branch: https://github.com/devinshoemaker/dev-to-git/tree/fix-http-429

Here's the code that defines the rate limit values in dev.to: https://github.com/thepracticaldev/dev.to/blob/5b62811c9874fa2770fe3f720abac1f80fe6279f/app/services/rate_limit_checker.rb#L11

I also noticed that the version of got used in this repo is a couple of major versions behind. There are some fixes for the retry option in more recent versions, but bumping up to even the next available version (10.0.0) causes a number of breaking changes that I'm not comfortable working through.

Hopefully, this give you all some more info to work off of. I might play around with this a bit more over the next couple days, but I'm not sure I'm familiar enough with the libraries and code base at this point.

[beeman]

When looking at some more DEV api usage, I came across this project, which seems to have a throttle for got.

[ApayRus]

is there a way to exclude files that weren't changed from sending to the server?

[ApayRus]

When looking at some more DEV api usage, I came across this project, which seems to have a throttle for got.

@beeman Do you use that project at the end?

[maxime1992]

hanged from sending to the server?

It does that already

[ApayRus]

Thank you for quick response. Why then Too many requests if updated only 1 page?

[maxime1992]

I never investigated that so I cannot answer that one.

To support my claim, you can have a look here: https://github.com/maxime1992/dev-to-git/blob/9704e9150c4bb6d98790f66be9d6184eaca1a93d/src/article.ts#L116-L123

As for the actual issue there, I'm not actively maintaining that repo. I'm ok reviewing PR but I don't have any more time to put into it I'm afraid so the fix on this one will have to come from the community

[maxime1992]

I just made a small speculative fix. Unlikely to fix but a small attempt as a quick fix...

https://github.com/maxime1992/dev-to-git/commit/e34dfafeca7d0842a9163ac3510b7af1fdb9a17e

[maxime1992]

Before: https://github.com/maxime1992/my-dev.to/actions/runs/5165970105/jobs/9305789445 :red_circle:

After: https://github.com/maxime1992/my-dev.to/actions/runs/5166090709/jobs/9305977103 :heavy_check_mark:

Looks like it may be enough for now!