Subhodip Roy issues

Repositories
Issues
Comments

Results 1 issues of


                                            Subhodip Roy

[Feat] Socket scale using ip-hash algo

This PR introduce the llm-server to scale horizontally by having multiple replicas. Fixes #625 **The approach** For only socket connections nginx will remember the request ip and redirect all relevant...