Retries & 429 Too Many Requests

[louwers]

When checking a lot of URLs of the same origin, it is very easy be rate-limited and get a 429 Too Many Requests response.

It would be neat if some retry logic could be added. The simplest implementation would simply look for a Retry After header. https://stackoverflow.com/a/63684455/1929678

[manuzhang]

@johnthagen WDYT?

[johnthagen]

Hmm, I don't see it as a bad idea but it could add a lot of complexity to the project?

@louwers Does this work as a workaround?

  - htmlproofer:
      raise_error: True
      raise_error_excludes:
        429: ['https://www.mkdocs.org/']

It feels a bit more flexible and if you know you are going to hit a certain URL a lot, you can simply count 429s as "passes"?

[louwers]

That solution does not work if you need to check a lof of URLS from the same domain.

But we have decided not to run external link validation on CI, but instead only check internal links. If someone else needs this, this can be re-opened.