Request: cache validated URLs

[rhagenson]

It was determined during a PR here, that in its current state mkdocs-htmlproofer will re-validate URLs it has seen before.

In order to maintain an upper bound on memory consumption, perhaps a new option should be added such as cache-size: 500 which will maintain a FIFO queue of the most recent 500 validated URLs. Each possible URL would then be checked for membership in the queue and, if present, the previous validation result repeated. My original idea a month ago was to only cache valid URLs, but I think caching invalid maybe is slightly improved -- although there is possible rationale in trying to re-validate on a set of possible error codes such as 429s, as we saw in the aforementioned PR.

@manuzhang Since you requested I open this issue, is there any additional information or question you have for me? (I apologize that I took a month to open this issue following your request.)

[manuzhang]

Thanks for opening this request and nice suggestion !

[manuzhang]

@rhagenson Please check out https://github.com/manuzhang/mkdocs-htmlproofer-plugin/pull/11

[johnthagen]

@manuzhang Should this issue be closed since #11 has been merged?

[manuzhang]

I leave it open because the cache size is hardcoded currently

[ssbarnea]

We need caching between runs and maybe even a git tracked cache file as the current implementation does showdown the execution by more than an order of magnitude. LRU cache does not help here.

[manuzhang]

We need caching between runs and maybe even a git tracked cache file

@ssbarnea Can you share your usage and elaborate on your idea?