flink-http-connector icon indicating copy to clipboard operation
flink-http-connector copied to clipboard

Implement rate limiting strategy

Open kristoffSC opened this issue 2 years ago • 1 comments

In Flink 1.16 a [Rate limiting API](https://flink.apache.org/2022/11/25/optimising-the-throughput-of-async-sinks-using-a-custom-ratelimitingstrategy/) was added to Flink Async Writer.

This can help to slow down rate of Http calls in case when HTTP endpoint is overwhelmed and timeouts or rejects HTTP requests. The API needed for this functionality is described here - https://flink.apache.org/2022/11/25/optimising-the-throughput-of-async-sinks-using-a-custom-ratelimitingstrategy/

Rate Limiting API should be available both in Table and Streaming API.

Probably, requests retry must be implemented first.

kristoffSC avatar Nov 05 '23 20:11 kristoffSC