JustinBeckwith
Request: Rate Limit
I love your CLI tool! I am truly missing just one thing - rate liming.
I use --concurrency 1 and even than, on a larger website, I get rate limited (our websites) usually after 10-15 seconds.
Some of them are on shared hosting so I have no way to whitelist/ignore my IP.
**It would be lovely to have an option to set "pause" inbetween the requests (in ms).
Ex.: --pause 500ms**
Thank you for this wonderful tool!
So this is a really tough one - ultimately a lot of websites are going to be very aggressive about this kind of thing. One of the ways I try to get around this is properly supporting 429 responses. For the websites where you're seeing this problem - do you know what status code they're returning? What's the behavior like when you get rate limited? An example site would be incredibly helpful!
Hello,
as for my shared hosting provider, it returns:
403 Forbidden
They use FailToBan I think, which rate limits me for about 60 seconds if requests in a set timeframe exceed some set ammount of requests.
After 60 seconds about, I get 200 Ok again right away - and the timeframe starts again.
It does not return 429 Too many requests before - it goes from 200 Ok to straight 403 Forbidden.