Include Tower feature 'load-shed'

[GrandmasterTash]

Feature Request

Please add the tower feature 'load-shed' to cargo.toml

Motivation

Under heavy load I'm seeing thread counts and memory explode (for some reason when the clients begin timing out).

This isn't prohibited by tower::limit::ConcurrencyLimitLayer as it backs-up the requests behind a semaphore.

I'm also not able to try the rate-limit layer because I'm using serve_with_shutdown() which requires layers implement Clone.

I would like to try the Tower feature tower::load_shed::LoadShed to see if this can alleviate my problem.

Proposal

Amend the dependency in Cargo.toml for tower to include "load-shed" in list of features.

[LucioFranco]

@GrandmasterTash if I understand correctly you'd like to use a time based rate limiter than a concurrency one? I think that makes sense. I think the rate limit layer currently is quite limited, we should improve this for sure.

[GrandmasterTash]

Pretty-much yes. I actually had this layer in-mind - but the rate limit layer could be useful as well.