Consider using nginx or vmauth for balancing read load to vmselects for cluster chart
Looks like balancing via default k8s service is not always optimal. During the benchmarks I found that vmselects are receving non equal number of queries. Especially, in cases when read queries have different execution time. This may result into excessive resource usage for some vmselects. Would be great to try using nginx for better balancing of the read load. Least-loaded strategy seems like a better fit here.
at least, Open source nginx can't aware resource consumption by backends. How can this be implemented ?
@cloud-66 we can try using vmauth instead, as it provides load-balancing feature out-of-box.
We can try to validate the assumption for vmauth to improve balancing by testing it on our playground.