[client-v2][discussion] Load-balancing on the client side.

[chernser]

Topic

Client-v2 implementation may connect to a single target host today. Client uses Apache HTTP client which has built-in connection pool. This set of properties is enough to handle many use-cases, because we assume that ClickHouse cluster is behind a load-balancer. Client-v1 has load-balancing on the client side and can handle failover to a backup node. This mechanism is complex because should track many moving parts.

Handling load balancing on a client side has some challenges:

• keep track of liveness of all nodes.
• load distribution requires very fine connection control.
• fair load distribution would need to know load of each pod.

External HTTP load balancer would work better:

• all known load-balancers have liveness check - no need to do it on client.
• proxy knows how many requests were sent to each node - so it is much easier to handle
• having a centralized proxy helps to control traffic to a DB. For example, if some application start acting bad - it can be easily disconnected from cluster.

Proxy would become a single point of failure, but it is lightweight and easy to restart than swarm of pods.

This issue is for the discussion. Please share your thoughts about pros, cons for both approaches. Thanks!