automq icon indicating copy to clipboard operation
automq copied to clipboard

[Enhancement] Fast retry on slow S3 request

Open SCNieh opened this issue 10 months ago • 0 comments

As per the best practice for optimizing S3 performance by AWS and Aliyun, a fast retry on slow request is expected to take a different path and respond quickly, which is able to improve the performance of catch-up read.

A possible design is to utilize a S3 client pool in which different client has an expected latency baseline for requests with different size, when the time elapsed since request exceeds the baseline by certain threshold, issue a fast retry instead of waiting for the former one

SCNieh avatar Apr 07 '24 10:04 SCNieh