wayback-machine-downloader icon indicating copy to clipboard operation
wayback-machine-downloader copied to clipboard

Make it async? [feature request]

Open mpr1255 opened this issue 2 years ago • 2 comments

I guess development here is no more -- but making this async would be really good/functional. Trying to backup a site with 175k links for example, I'm on URL 80 after a minute. Not really feasible.... I'm going to do something in python but it will be very basic and not a really great CLI like this.

mpr1255 avatar Jun 19 '23 04:06 mpr1255

you can use -c to download multiple files at a time, it will be way faster.

e.g. -c 20 means 20 files at one time

EvanNotFound avatar Aug 19 '23 00:08 EvanNotFound

A side note: the internet archive has implemented a rate limit feature which probably breaks all mutli-threaded downloads, like -c, since it breaks single threaded downloading as well without substantial delays between requests. See pull request #268 and #266 for more details

https://archive.org/details/toomanyrequests_20191110

sww1235 avatar Nov 20 '23 03:11 sww1235