Load-balancing / auto-scaling for LLM serving on Azure

[VictorOdede]

[internot169]

Hi @VictorOdede, I'd like to try this issue. Could you please provide some more information?

[kaushikdaiv7]

I would love to work on this @VictorOdede , can you share details for this issue

[lucylililiwang]

Hi! Vik, Can I also please work on this issue? Thank you!

[lucylililiwang]

When we are taking care of the Load-balancing, is it alright for us to do Azure Kubernetes Service (AKS) along with Horizontal Pod Autoscaler (HPA) and Kubernetes Ingress Controller for load balancing? Thank you!

[lucylililiwang]

sorry, when we setting up the Kubernetes cluster, which Kubernetes version should we choose? Thank you!