Local LLM as backend for DemoGPT agent

[paluigi]

Is your feature request related to a problem? Please describe. Using local LLMs instead than OpenAI API as backend

Describe the solution you'd like Create a DemoGPT agent from a locally available model (ideally, a quantized Llama2 model via llama-cpp-python

Describe alternatives you've considered If that' s already possible, a guide or some instruction about how to do it would be greatly appreciated!

Additional context NA

[melih-unsal]

Hi @paluigi,

Thank you for highlighting this feature request. We truly value your feedback and are always eager to improve DemoGPT based on our community's suggestions.

At present, our primary focus is enhancing DemoGPT's capabilities by adding more tools. That said, integrating local models is definitely on our roadmap, and Llama2 is indeed at the top of our list for such integrations.

We appreciate your input and dedication to the growth of DemoGPT. 🙏 Stay tuned for updates!

[wusiyaodiudiu]

Hi @paluigi You might consider referring to the implementation here, which utilizes Fastchat to encapsulate other open-source models for invocation through the OpenAI API. I've employed the chatglm-6b model here, though I'm uncertain whether it supports the llama-cpp model.

Link: https://github.com/chatchat-space/Langchain-Chatchat/blob/master/server/llm_api.py

If necessary, I can submit the code to GitHub. @melih-unsal

[paluigi]

Thanks @wusiyaodiudiu , I will have a look to your repo!

[wusiyaodiudiu]

Hi @paluigi I have already forked this project and committed the relevant implementation code for the local LLM,This time, I only committed the part related to the local LLM, so there may be some minor errors. However, I believe the implementation approach is relatively easy to understand and extend.

Newly added files: demogpt/model_config.py demogpt/server/llm_api.py Modified files:" demogpt/app.py demogpt/model.py