[Bug] [0.4.1] 使用vllm启动时报错

[lzhshe]

trafficstars

Search before asking

• [X] I had searched in the issues and found no similar issues.

Operating system information

Linux

Python version information

3.10

DB-GPT version

main

Related scenes

• [ ] Chat Data
• [ ] Chat Excel
• [ ] Chat DB
• [ ] Chat Knowledge
• [ ] Model Management
• [ ] Dashboard
• [ ] Plugins

Installation Information

• [X] Installation From Source

• [ ] Docker Installation

• [ ] Docker Compose Installation

• [ ] Cluster Installation

• [ ] AutoDL Image

• [ ] Other

Device information

-GPU count :1 -GPU memory 24G

Models information

LLM:baichuan-13b

What happened

初始化项目，执行pip install -e ".[default]" 正常启动和问答，然后执行pip install -e ".[vllm]"，设置MODEL_TYPE=vllm，启动报错如下： INFO 11-22 17:01:16 llm_engine.py:72] Initializing an LLM engine with config: model='/root/autodl-tmp/model-v2/models/Baichuan-13B-Chat', tokenizer='/root/autodl-tmp/model-v2/models/Baichuan-13B-Chat', tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, quantization=None, seed=0) 2023-11-22 17:01:16 autodl-container-22b9119c3c-481f965c pilot.model.cluster.worker.manager[31073] ERROR Error starting worker manager: model baichuan-13b@vllm(172.17.0.4:6006) start failed, 'BaichuanTokenizer' object has no attribute 'sp_model' 2023-11-22 17:01:16 autodl-container-22b9119c3c-481f965c asyncio[31073] ERROR Task exception was never retrieved future: <Task finished name='Task-3' coro=<_setup_fastapi..startup_event..start_worker_manager() done, defined at /root/autodl-tmp/model-v2/pilot/model/cluster/worker/manager.py:759> exception=SystemExit(1)> Traceback (most recent call last): File "/root/autodl-tmp/model-v2/pilot/model/cluster/worker/manager.py", line 761, in start_worker_manager await worker_manager.start() File "/root/autodl-tmp/model-v2/pilot/model/cluster/worker/manager.py", line 577, in start return await self.worker_manager.start() File "/root/autodl-tmp/model-v2/pilot/model/cluster/worker/manager.py", line 114, in start raise Exception(out.message) Exception: model baichuan-13b@vllm(172.17.0.4:6006) start failed, 'BaichuanTokenizer' object has no attribute 'sp_model'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "uvloop/loop.pyx", line 474, in uvloop.loop.Loop._on_idle File "uvloop/cbhandles.pyx", line 83, in uvloop.loop.Handle._run File "uvloop/cbhandles.pyx", line 63, in uvloop.loop.Handle._run File "/root/autodl-tmp/model-v2/pilot/model/cluster/worker/manager.py", line 764, in start_worker_manager sys.exit(1) SystemExit: 1 请问是环境哪里不对吗

What you expected to happen

希望正常启动

How to reproduce

1、pip install -e ".[default]" 2pip install -e ".[vllm]" 3、 python pilot/server/dbgpt_server.py --port 6006

Additional context

No response

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

[fangyinc]

可以参考这个解决一下。

[wangzaistone]

可以参考这个解决一下。 After pip install transformers==4.33.1 tokenizers==0.13.3 ,it will report vllm version eror ,like this :

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
vllm 0.2.2 requires transformers>=4.34.0, but you have transformers 4.33.1 which is incompatible.
Successfully installed tokenizers-0.13.3 transformers-4.33.1 

截止20231127-11h , 更改完 “ tokenization_baichuan.py ，把 super() 修改到最后执行”后，用transformers==4.33.1，baichuan2的错误不再报，但是会导致vllm 报关于“ cannot import name 'MistralConfig' from 'transformers'” 的错误[https://github.com/vllm-project/vllm/issues/1458]。 如果将升级到4.34.1 “ pip install transformers==4.34.1” ，“'BaichuanTokenizer' object has no attribute 'sp_model' 的错误就不能解决。

[eigen2017]

super改了不生效，系统不知道从哪里拷了一份改之前的，放到了cache下面，然后就加载cache下面的，我也是vllm加载baichuan2-13b 尝试了vllm启动时--tokenizer到模型的文件夹，依然不行

[eigen2017]

super改了不生效，系统不知道从哪里拷了一份改之前的，放到了cache下面，然后就加载cache下面的，我也是vllm加载baichuan2-13b 尝试了vllm启动时--tokenizer到模型的文件夹，依然不行

是我没看清，修改模型文件夹中的就行了。注意保存

[github-actions[bot]]

This issue has been marked as stale, because it has been over 30 days without any activity.

[github-actions[bot]]

This issue bas been closed, because it has been marked as stale and there has been no activity for over 7 days.