vllm-project
[RFC]: stormservice support vllm and sglang multiple-node deployment
Currently, we lack a unified multi-machine deployment solution. Raycluster currently supports multi-machine deployment using vllm, but does not support multi-machine deployment using sglang. The LeaderWorkerSet recommended by sglang lacks flexibility, so it is possible to use stormservice to support both types of inference.
Sure. I think we just need to add a vLLM multi-node example using stormservice. is that what you are looking for?
Sure. I think we just need to add a vLLM multi-node example using stormservice. is that what you are looking for?
Firstly, podGroupSize should be used to enable smoother deployment of distributed inference for vllm. Secondly, the node rank requirements for sglang distributed inference should also be met.
@ying2025 got you. I will ask some contributors to write some concrete examples and explanation. the podGroupSize is a little bit confusing to most users. We will make sure this is clearly explained this time.