remote-apis-sdks icon indicating copy to clipboard operation
remote-apis-sdks copied to clipboard

Introduce heap balancer for the gRPC connection pool

Open mrahs opened this issue 6 months ago • 0 comments

We observed increased latencies in builds that use the round-robin balancer. It is possible that the load is not uniformly spread across the pool, particularly around streams. This could cause client-side queuing on hot connections which may explain the increased latencies.

The heap-based balancer keeps track of the number of invocation and streaming requests per connection and selects the least used connection. One big caveat is that the pool is not aware of active connections. The number of stream requests is not necessarily the number of active streams on the connection. Without that knowledge, it is still possible for some connections to be used more than others. E.g. if long streaming requests accumulate on the same connection.

mrahs avatar Jul 29 '24 21:07 mrahs