429 Too Many Requests

[lostact]

Hi, Recently archive.org has put a rate limit so making too much requests per minute will result 429 error and you have to wait some time to make more requests. So if you use concurrency (specialy above 4) some pages will be missing.

The issue can be easly solved by adding a sleep(n) then retry in download_file function if the http_error code is equal to 429. I don't know the rate limit details so i cant tell how much seconds we have to wait.

[hockeybro12]

How did you fix this? I installed the package using gem install. How can I add that sleep?

[AstroDan]

@lostact could you post your solution. No Ruby experience here!

[dhouck]

Or make a PR

[PuccamiteTech]

For starters, don't use a value higher than three with the -c switch. That should fix your problem! :)

[baob]

Did anyone come up with a fix for this ? Or alternatively can anyone confirm keeping to -c 3 or below a reliable solution ?

[dbsanfte]

I never hit rate limit with -c 4.