Import Errors occurring even when dependencies are installed

[david-vectorflow]

I am unable to create a Runtime with sglang as follows runtime = sgl.Runtime(model_path=MODEL_DIR, tokenizer_path=MODEL_DIR). It throws the error below:

ImportError: cannot import name '_set_default_torch_dtype' from 'vllm.model_executor.model_loader' 
(/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/__init__.py)

I am running this on an A100 with the Nvidia PyTorch base image recommended by the Flash Attention repo ("nvcr.io/nvidia/pytorch:24.03-py3"). I have confirmed via print statements that all the dependencies are installing in the desired location and the files are actually there. I have tried installing sglang both with pip and from source and get the exact same error.

I noticed that not every package in the file structure has an __init__.py file, which can cause issues. Perhaps the version of PyTorch in the base image conflicts somehow with what sglang expects? Aside from that, I am not sure why this is happening. But given how simple this code is, I suspect there is an issue with the library, not the machine.

[nikita-lukashevich1]

Look here: https://github.com/sgl-project/sglang/issues/391

[david-vectorflow]

Thanks! Looks like it still isn't solved though?

[github-actions[bot]]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.