sotoki
sotoki copied to clipboard
Stop relying on online archive.org Sites.xml
Now that archive.org is down, it becomes obvious that the scraper is down as well despite the fact that we have mirrored all dumps. And the reason is a bit sad: we did not mirrored Sites.xml
and we are fetching it from online ... which is not online anymore.
I think we must mirror Sites.xml as well, and use the Sites.xml for our S3 bucket.