vLLM import error

[jlin816]

I'm getting the following import error:

sgl ➜ export CUDA_VISIBLE_DEVICES=4; python -m sglang.launch_server --model-path meta-llama/Llama-2-7b-chat-hf --port 30000
Traceback (most recent call last):
  File "/home/jessy/.miniconda3/envs/sgl/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/jessy/.miniconda3/envs/sgl/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/jessy/projects/sglang/python/sglang/launch_server.py", line 3, in <module>
    from sglang.srt.server import ServerArgs, launch_server
  File "/home/jessy/projects/sglang/python/sglang/srt/server.py", line 56, in <module>
    from sglang.srt.managers.router.manager import start_router_process
  File "/home/jessy/projects/sglang/python/sglang/srt/managers/router/manager.py", line 9, in <module>
    from sglang.srt.managers.router.model_rpc import ModelRpcClient
  File "/home/jessy/projects/sglang/python/sglang/srt/managers/router/model_rpc.py", line 24, in <module>
    from sglang.srt.managers.router.model_runner import ModelRunner
  File "/home/jessy/projects/sglang/python/sglang/srt/managers/router/model_runner.py", line 15, in <module>
    from vllm.model_executor.model_loader import _set_default_torch_dtype
ImportError: cannot import name '_set_default_torch_dtype' from 'vllm.model_executor.model_loader' (/home/jessy/.miniconda3/envs/sgl/lib/python3.10/site-packages/vllm/model_executor/model_loader/__init__.py)

It looks like _set_default_torch_dtype no longer exists in vllm: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/model_loader/__init__.py

I've tried both pip install sglang[all] and installing from source. My versions are sglang==0.1.14 and vllm==0.4.1.

[m0g1cian]

As of now, you need to downgrade your vllm version to 0.3.3

[dmilcevski]

I have the same error. But as far as I can see version 0.3.3 of vllm doesn't have model_loader in vllm.model_executor.

[epellis]

I also have this error, looks like that function got removed from vllm in this pr when they refactored model loading: https://github.com/vllm-project/vllm/pull/4097

It's defined here: https://github.com/vllm-project/vllm/blob/05434764cd99990035779cf9a4ed86623b528825/vllm/model_executor/model_loader.py#L22-L28.

Looks like some other functions also got removed in the same PR like hf_model_weights_iterator that are also used to load all the models.

[david-vectorflow]

Attempting to downgrade to vllm 0.3.3 causing the installation of the most recent version of sglang to hang indefinitely

[ddemillard]

I too am seeing this just hang indefinitely if I downgrade. Are there any workarounds?

[ChloeL19]

I too am seeing this just hang indefinitely if I downgrade. Are there any workarounds?

My setup is vllm==0.3.3, transformers==4.38, sglang==0.1.14 My sglang script also hangs

[zhyncs]

Please try the latest version https://pypi.org/project/sglang/