Stop on Rate limit flag

[tarunKoyalwar]

Description

Usually lot of subdomains have rate limits while bruteforcing for directories gobuster should stop if server responds with 429 status code but gobuster does not stop or exit and runs until wordlist which is meaning less and sometimes depending on implementation server might blacklist IP address

Solution

This should be implemented directly however if not at least a --stop-at-rtl flag should be available

I would love to contribute if agreed

[antoninoLorenzo]

I also encountered this issue, however rather than stopping I think a throttling mechanism would be more useful. If you are going to fix this you should consider:

• throttling after 429 (your case)
• throttling by default, it would be useful if something more advanced of a rate limit is in place;
• edge cases such as X-RateLimit-Limit header (common in APIs)

[ethanannane]

I’m new to open source, but I’d be willing to have a go at implementing this if it’s something people would like. I’m not entirely sure whether this should be the default behavior or live behind a flag. Along with stopping on rate limit, I agree throttling could be useful too, and might make more sense as the default. Would you be open to me having a try at this, and perhaps pointing me in the right direction if I get stuck?

[firefart]

feel free to submit a pr against the dev branch. the stop on rl should be disabled by default and enabled by a cli flag

[ethanannane]

ok i have had a go at it and so far i have it implemented for dir in this fork if someone could look at what ive done so far and tell me if im going in the right direction. thanks!