AgentGPT
AgentGPT copied to clipboard
reworkd
🤖 Investigate utilizing local models
This is a big undertaking. As a user, I'd like to be able to use my own local models to not be crippled by limitations of OpenAI models. The opensource model scene is getting huge and we want to support this as best as possible. The issue is, given we are a product first and foremost, connection to local models must work in both prod and dev and must be as simple as possible to setup.
One option is WindowAI: https://twitter.com/xanderatallah/status/1643356106670981122. This is an extension that will allow a site to connect to a model locally.
Before implementing anything, we should first investigate possibilities and report back in this issue.
@asim-shrestha I would be interested in assisting with this issue.
On initial analysis, it appears that window.ai uses a combination of Python + Flask to host a model local to the server, provided by Flask. The repo includes an example using a model ftrom hugging face, imported locally, and being used by the server to support a completions endpoint.
In my opinion, models such as the following could also be investigated for suitability as candidates for use locally with AgentGPT:
-
Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%*Â ChatGPT Quality, with vicuna-13b-4bit at huggingface.
-
Alpaca, with variants such as Pi3141/alpaca-7b-native-enhanced at huggingface. There is an interesting blog on How to Run a ChatGPT-Like LLM on Your PC Offline.
I'm not a specialist, but the ideal for me would be to be able to mix AgentGPT and GPT4All so that I could use any model locally, without having to spend money with OpenAI to use ChatGPT. Some models perform better than ChatGPT in certain areas, and being able to use AgentGPT on them would be a huge leap forward.
@psgit @AgimaFR, haven't looked too closely at the local model scene. Are there any that expose an API that we can download directly from docker-compose at this point? I think GPT4All has an installer process?
Won't be able to use them in prod but would be a huge win for local installations of AgentGPT. (Though I question their performance capabilities / abilities to output structured JSON)