Rate Limit By Applying Backpressure and Buffering

[adisoluto]

Some consumers have a limited ability to handle incoming requests. We would like to be able to protect these consumers from traffic spikes, thus, rate limiting DQD and applying backpressure. We would like to have the ability to control how many requests are consumed and processed per time unit, while buffering the events in the queue.

The offered solution is to consume the messages at a fixed rate. For example - is the rate is 1request/second, dqd will sleep for 1 sec and consume 1 message.

Note that this solution might delay the consumption and even if the limit hasn't been exceeded.

This use case is intended to handle traffic spikes only. If the producer is producing a higher rate of events regularly (for example 40requests/second), the buffer will never finish. Using this feature requires monitoring on the source.

Points to consider: multiple DQD instances that need to share the load.

[CHarnel]

A concrete use case is: events coming in batches via queue and we want to pipe them to an http endpoint which is rate limited. So we want to consume messages from the queue at a specified rate (e.g. 20 per second).

[Yshayy]

There's a support for fixed rate: https://github.com/Soluto/dqd/blob/master/config/config.go#L124 It limits the amount of concurrency for maximum amount of processing units. (for example, only one processing unit). Of course, it won't solve the problem if even a single processing unit is too fast.

In that case, it can be slowed by better error back off policy (which can be error based, like for 503), or alternatively, set minimum time to processing job (which is similar to rps per processing unit)

Fixed the link

[Yshayy]

Another way to implement it, is creating input/output source that does rate-limiting and create a more complex pipe, but scaling processing unit can be a bit tricky

[adisoluto]

I updated the proposed solution to fixed rate consumption.