LLM-VM icon indicating copy to clipboard operation
LLM-VM copied to clipboard

Published 20 hours ago • verified publisher icon anarchy-ai

Load-balancing / auto-scaling for LLM serving on AWS

Open VictorOdede opened this issue 1 year ago • 3 comments

VictorOdede avatar Oct 24 '23 10:10 VictorOdede

What did you have in mind here? when you refer to autoscaling, are you referring to horizontal scalability, or parallelism across a single host?

horahoradev avatar Oct 28 '23 01:10 horahoradev

@horahoradev I was actually referring to horizontal scaling of LLM instances

VictorOdede avatar Oct 31 '23 15:10 VictorOdede

Hi, can I please work on this issue? Thank you!

lucylililiwang avatar Feb 13 '24 16:02 lucylililiwang