Sticky Load Balancing during inference not hitting all the Triton Servers sitting behind the Load Balancer

[animikhaich]

I have an architecture that looks like this:

The main issue is that the load balancer is not distributing the traffic across the different server VMs. Instead at any given time I only see 40-50% of the VMs under full load being used. I am closing the gRPC client after each request using client.close(). However, it seems that session is still sticky, which is likely the root cause.

I am using Azure Load Balancer.

Would love to know if anyone has faced this issue and resolved it. If so, how?

[patches11]

+1 this is a big issue, how has this not been responded to at all?