Find a way to deal with 429 Too Many Requests

[jyn514]

When validating a lot of URLs with --check-http, it's common to get '429 Too Many Requests'. It would be great to be able to ignore these. Alternatively, maybe cargo-deadlinks could add a rate-limit option that avoids getting the error in the first place.

[ayazhafiz]

This is a hard thing to fix in a general way because different rate limits (if they have any at all).

[zummenix]

I wonder if https://github.com/deadlinks/cargo-deadlinks/pull/63 mitigates the problem.

[jyn514]

I tested this out by running deadlinks book on https://github.com/rust-lang/rustc-dev-guide/. As expected, it gave a bunch of 429 errors, so unfortunately #63 didn't help (although I haven't tested with an earlier version of deadlinks, so it might be fewer 429s than before).

This is a hard thing to fix in a general way because different rate limits (if they have any at all).

Right, this is why I think --rate-limit should be passed by the user and not hard-coded by deadlinks. Alternatively, deadlinks could give a warning but not fail the request.