tonic
tonic copied to clipboard
Add support for load-shedding.
Add an option to enable load shedding for the service constructed for the connection.
Motivation
In the server we are implementing we want to load-shed when the service is overloaded, and we want to limit the number of concurrent requests per connection. We use tower::load_shed layer for load shedding, but this only applies to the entire service.
If we additionally apply the per-connection concurrency limit the issue is that this limit is typically hit first and so the service responds with "not ready". As a result the load-shed layer of the service is never reached meaning that requests just queue up.
Solution
The solution is to optionally add a load-shed layer to the per-connection service constructed by tonic.