internetarchive
Dead hosts component
In warcprox and in the WBM live web checker we have the concept of "dead hosts". These are hosts which always deny our connections.
https://github.com/internetarchive/warcprox/blob/f31143183490b29182ba1d11ca908d18b5b5e572/warcprox/mitmproxy.py#L397
When a host:port denies N number of connections, we add them to the dead hosts set in memory. For every new URL we always check if they belong to a dead host to avoid doing the request altogether. This way, we skip a lot of invalid connections and useless waiting for network timeout.
The dead host cache can be refreshed every hour / day according to the implementation. Zeno could keep the dead hosts in internal memory (it will not be common across all Zeno instances).
Good idea, but considering the side effects it could have, I'd tempted to have it off by default? Not sure. @NGTmeaty ?
I think it would be fine to leave off by default. If there are specific hosts or IPs that are consistently not letting us connect or denying our connections, it would likely be in our best interest to ratelimit those attempts to save resources.
Maybe this ties into a "optimized" Zeno config or list of arguments that we could distribute, making it easier for people to get started with Zeno.
