tonic icon indicating copy to clipboard operation
tonic copied to clipboard

Add support for load-shedding.

Open abizjak opened this issue 5 months ago • 0 comments

Add an option to enable load shedding for the service constructed for the connection.

Motivation

In the server we are implementing we want to load-shed when the service is overloaded, and we want to limit the number of concurrent requests per connection. We use tower::load_shed layer for load shedding, but this only applies to the entire service.

If we additionally apply the per-connection concurrency limit the issue is that this limit is typically hit first and so the service responds with "not ready". As a result the load-shed layer of the service is never reached meaning that requests just queue up.

Solution

The solution is to optionally add a load-shed layer to the per-connection service constructed by tonic.

abizjak avatar Jan 30 '24 21:01 abizjak