[Bug]issues with separate deployment of the base large model and DBGPT project

[zhaoyinjiang9825]

Search before asking

• [X] I had searched in the issues and found no similar issues.

Operating system information

Linux

Python version information

=3.11

DB-GPT version

main

Related scenes

• [X] Chat Data
• [ ] Chat Excel
• [ ] Chat DB
• [ ] Chat Knowledge
• [ ] Model Management
• [ ] Dashboard
• [ ] Plugins

Installation Information

• [X] Installation From Source

• [ ] Docker Installation

• [ ] Docker Compose Installation

• [ ] Cluster Installation

• [ ] AutoDL Image

• [ ] Other

Device information

GPU:8 GPU_Memery:24GB

Models information

LLM：Qwen_72B_SFT微调大模型 Embedding_model:text2vec

What happened

请问底座大模型和DBGPT项目分开部署，大模型部署在一台机器上，DBGPT部署在另一台机器上，这种情况是否有对应的解决方案

What you expected to happen

麻烦指导一下，源码需要改哪些地方

How to reproduce

无

Additional context

No response

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

[Aries-ckt]

try cluster deploy mode https://docs.dbgpt.site/docs/installation/model_service/cluster

[zhaoyinjiang9825]

请问是否可以添加用户自己的模型代理，除了官方提供的那几个代理以外，我想将自己的微调模型做成代理供dbgpt使用，这种方式有解决方案吗？

[zhaoyinjiang9825]

@Aries-ckt

[Aries-ckt]

yeah, you should write your LLM service adaptor.

[Dd88664185]

try cluster deploy mode https://docs.dbgpt.site/docs/installation/model_service/cluster 404 打不开

[zhaoyinjiang9825]

我作了如下工作： （1）参照baichuan_proxy创建了dbgpt\model\proxy\llms\railway.py，输入输出都和baichuan.py一样。 （2）在dbgpt\model\adapter\proxy_adapter.py，添加了如下代码: (3)dbgpt\configs\model_config.py和dbgpt_private\config.py都做了相应的修改 第一个： 第二个：

修改.env文件，换成自己指定的代理但还是跑不起来报如下错误： 请问除了这些，我还需要做哪些工作？@Aries-ckt

[Dd88664185]

是的，您应该编写 LLM 服务适配器。

你好 我集群部署也遇到了问题 我modesl list 是可以查看到llm模型，并且前端页面也可以看到其他服务器推送过来的模型，但使用时候就报错，请问这个该怎么解决

[github-actions[bot]]

This issue has been marked as stale, because it has been over 30 days without any activity.

[github-actions[bot]]

This issue bas been closed, because it has been marked as stale and there has been no activity for over 7 days.