Stas Bekman

icon indicating a website link https://stasosphere.com/machine-learning/

icon company Stasosphere Online Inc. / icon location BC, Canada icon location Toolmaker. Author. Software creator, optimizer and harmonizer. Makes things work. Current domains: LLM/Training/Inference/Scalability/Machine Learning

Results 664 comments of Stas Bekman

[Usage]: How to use Multi-instance in Vllm? (Model replication on multiple GPUs)

with online setup yes it'd work, but this is an offline recipe please read https://github.com/vllm-project/vllm/issues/6155#issuecomment-2261755228

[Usage]: How to use Multi-instance in Vllm? (Model replication on multiple GPUs)

@njhill didn't post the update since his last note https://github.com/vllm-project/vllm/issues/6155#issuecomment-2261779014, so I wasn't able to validate it - do you have the PR link - would be happy to re-test.

[Usage]: How to use Multi-instance in Vllm? (Model replication on multiple GPUs)

Thank you for the update, @njhill - there was a progress and it no longer hangs in the init of the 2nd model, but now it hangs in: ``` INFO...

[Usage]: How to use Multi-instance in Vllm? (Model replication on multiple GPUs)

Thank you, @njhill - my example can serve as a repro. And of course we want to have more than one `llm` object co-exist and not run those sequentially only...