grab-site icon indicating copy to clipboard operation
grab-site copied to clipboard

Automatically slow down for a domain on 429 Too many requests

Open ivan opened this issue 7 years ago • 1 comments

Write a better default wait_time hook that does this; see https://github.com/ludios/grab-site/issues/59#issuecomment-343104125 for an example.

ivan avatar Nov 09 '17 10:11 ivan

Possible implementation strategy:

Implement #59 so that the user can easily adjust delays on a per-domain basis.

For each 429 response, add (# of connections being used * 1 second) to the wait time for that domain in the delay_regexps file e.g. 2000 ^https?://foo\.tld/, then rewrite delay_regexps.

ivan avatar Nov 09 '17 10:11 ivan