minoru-fediverse-crawler
minoru-fediverse-crawler copied to clipboard
Stop checking domains that have been dead for a year
I'm not so sure about this one anymore. If we stop checking the domain, and it comes back alive, we still won't check it.
We could drastically reduce the rate at which such domains are checked. Currently we do that once a week, which actually isn't that bad?
Alternative design would be to check if the domain is still on somebody's instances list, and if it's not, then we can stop checking it. This relies on everybody else removing the domain from their lists though, which IMHO Is not a good bet.