vllm-project
How to load a local model file?
I want to load a local model which has the same file with the files downloaded from huggingface. However, right now this repository seems to only support load from website.
@BUAADreamer @liulfy The model argument in LLM or api_server can also take the path to your local directory that contains the weight files.
But this code seems to request huggingface website
from vllm import LLM, SamplingParams
prompts = [
"你好，请介绍一下你自己",
"中国的定义是什么？"
]
sampling_params = SamplingParams(temperature=0.01, top_p=0.01)
llm = LLM(model="/path/to/llama")
@BUAADreamer Thanks for providing the example. It should not use the remote HF repo if the path is valid. Could you try this and let us know if it works?
from transformers import AutoConfig
config = AutoConfig.from_pretrained("your/path/to/llama")
If this doesn't work, then it means your path is somehow incorrect (or it's a bug in HF).
I use Build from source and run seccessfully. Thanks! But the Install with pip seems to have some problems
Indeed. I built from source: https://github.com/vllm-project/vllm/releases/tag/v0.1.1, and this problem solved.
https://vllm.readthedocs.io/en/latest/getting_started/installation.html just use the Build from source in this website and run like above, if it only report error, then set tokenizer_mode='slow'
Indeed. I built from source: https://github.com/vllm-project/vllm/releases/tag/v0.1.1, and this problem solved.
Yes, this version works