S3Scanner icon indicating copy to clipboard operation
S3Scanner copied to clipboard

Published 20 hours ago • verified publisher icon sa7mon

Investigate rate limiting

Open sa7mon opened this issue 1 year ago • 2 comments

Haven't run into 429's yet in testing, but if we crank the threads up enough it's bound to happen.

Probably test against AWS and see what happens. If we start hitting, implement exponential backoff.

Maybe the SDK client already has this? https://docs.aws.amazon.com/sdk-for-go/api/aws/client/#DefaultRetryer

sa7mon avatar Jul 11 '23 12:07 sa7mon

It's not 429, if you reached 100k requests, amazon will return 404 for every requests

Xifeng2009 avatar Aug 15 '23 12:08 Xifeng2009

Do you know what the time frame is? 100k in 1 hour?

On Tue, Aug 15, 2023 at 7:16 AM, Xifeng2009 @.***(mailto:On Tue, Aug 15, 2023 at 7:16 AM, Xifeng2009 < wrote:

It's not 429, if you reached 100k requests, amazon will return 404 for every requests

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

sa7mon avatar Aug 15 '23 17:08 sa7mon