black-mirror icon indicating copy to clipboard operation
black-mirror copied to clipboard
verified publisher icon T145

[feature]: Back up lists to the Internet Archive

Open T145 opened this issue 2 years ago • 3 comments

Contact Details

No response

What's your idea?

Instead of archiving lists to a local submodule, use the Internet Archive's API and libraries to backup lists and fall back to that URL if the original can't be accessed.

Code of Conduct

  • [X] I agree to follow this project's Code of Conduct

T145 avatar Feb 15 '24 18:02 T145

Prepend list URLs with https://web.archive.org/save/. Submitted 200 response means it gets saved successfully.

429: Save Page limits 15 requests per minute and will rate limit otherwise. This project should be fine. 509: Active session limit reached. Again this project should be fine.

T145 avatar Apr 09 '24 15:04 T145

Could detach the process so building can run alongside archival.

https://unix.stackexchange.com/questions/269805/how-can-i-detach-a-process-from-a-bash-script

The best plan is likely to perform archival before downloading the lists, catch any download errors, and retry those downloads with IA links.

T145 avatar May 01 '24 16:05 T145

(cb00c3f)

T145 avatar Jul 07 '24 00:07 T145