Alexey Gorbatovski

icon company ITMO University icon location Moscow icon location AI Researcher

Results 2 comments of Alexey Gorbatovski

[Usage]: How to use Multi-instance in Vllm? (Model replication on multiple GPUs)

Are there any updates?

[Bug]: client socket has timed out while trying to connect to GPU node, when initializing DeepSeek R1 in ray vllm serving

Same error while creating two vllm.LLM instances on more than 2 GPU ``` vllm_model = vllm.LLM( model=model_inference_settings.model_settings.model_path.absolute().as_posix(), **engine_settings.dict(), ) sft_model = vllm.LLM( model=model_inference_settings.sft_settings.model_path.absolute().as_posix(), **sft_engine_settings.dict(), ) ```