automq
automq copied to clipboard
[Enhancement] Fast retry on slow S3 request
As per the best practice for optimizing S3 performance by AWS and Aliyun, a fast retry on slow request is expected to take a different path and respond quickly, which is able to improve the performance of catch-up read.
A possible design is to utilize a S3 client pool in which different client has an expected latency baseline for requests with different size, when the time elapsed since request exceeds the baseline by certain threshold, issue a fast retry instead of waiting for the former one