wayback-machine-downloader
wayback-machine-downloader copied to clipboard
Make it async? [feature request]
I guess development here is no more -- but making this async would be really good/functional. Trying to backup a site with 175k links for example, I'm on URL 80 after a minute. Not really feasible.... I'm going to do something in python but it will be very basic and not a really great CLI like this.
you can use -c to download multiple files at a time, it will be way faster.
e.g. -c 20 means 20 files at one time
A side note: the internet archive has implemented a rate limit feature which probably breaks all mutli-threaded downloads, like -c, since it breaks single threaded downloading as well without substantial delays between requests. See pull request #268 and #266 for more details
https://archive.org/details/toomanyrequests_20191110