Introduce backpressure when rate limits are hit

[ellisjoe]

What happened?

Currently a client is able to enqueue an unbounded number of requests into the Dispatcher.

What did you want to happen?

One a certain number of requests are enqueued, we should either start throwing 429s back out of the client, or start blocking the threads enqueueing requests to introduce some form of backpressure upstream.

[j-baker]

I don't believe the client is able to do this without the concurrency limit rising to a higher number than the per-host limit. If you'd like, we should set a 'max concurrency' on the concurrency limiter, which at least avoids the noisy neighbour states?

[ellisjoe]

This happens when the incoming request rate exceeds the concurrency limit, regardless of where the limit is in relation to the per-host limit. This won't prevent the concurrency limit from exceeding the per-host limit, but it'll push back on clients when that does happen.

[stale[bot]]

This issue has been automatically marked as stale because it has not been touched in the last 60 days. Please comment if you'd like to keep it open, otherwise it'll be closed in 7 days time.