Is it possible to replace Chat-GPT with local offline model? For example, gpt4all, LLaMa, etc

[Neronjust2017]

trafficstars

Duplicates

• [X] I have searched the existing issues

Summary 💡

Is it possible to replace Can chat-GPT with local offline model? For example, gpt4all, LLaMa, etc

Examples 🌈

No response

Motivation 🔦

No response

[mendeltem]

An excellent suggestion would be to use local models. AUTOGPT can utilize these models to train individuals to become experts and also develop several agents simultaneously. Additionally, it can train and facilitate communication between these models.

[arronKler]

just replace the request to openai with your own models service in llm_utils.py. But the embedding part may need to keep using openai's embedding api.

[Neronjust2017]

maybe I should ask Auto-GPT to analyse itself and figure out how to replace openai api and chat-gpt with local LLM models, lol.

[zachary-kaelan]

Depends on if you have a GPU cluster and 64 GB or so of RAM to run anything comparable at a reasonable speed. You also gotta calculate impact on your electric bill.

[matthewniemeier]

Depends on if you have a GPU cluster and 64 GB or so of RAM to run anything comparable at a reasonable speed. You also gotta calculate impact on your electric bill.

There's a ton of smaller ones that can run relatively efficiently. Glance the ones the issue author noted.

GPT4All | LLaMA

[Neronjust2017]

There's a ton of smaller ones that can run relatively efficiently. Glance the ones the issue author noted. you are right. https://github.com/nomic-ai/pyllamacpp provides offilcal supported python bindings for LLaMA and gtp4all. Maybe I can use the pythonic API in llm_util.py to replace OpenAI-related API.

[matthewniemeier]

There's a ton of smaller ones that can run relatively efficiently. Glance the ones the issue author noted. you are right. https://github.com/nomic-ai/pyllamacpp provides offilcal supported python bindings for LLaMA and gtp4all. Maybe I can use the pythonic API in llm_util.py to replace OpenAI-related API.

New repo to browse just dropped fam 😅👌

[Neronjust2017]

just replace the request to openai with your own models service in llm_utils.py. But the embedding part may need to keep using Openai's embedding api.

Can I avoid using OpenAI's embedding APIs? Because the network connection to OpenAI can not be always established successfully, I want to be totally offline.

[talvasconcelos]

I wouldn't mind if the agents were from ChatGPT, the rest (i assume the fast_llm is the one the user interacts with) would be a local gpt4all, for example!

[mendeltem]

It would be really have a mutliple online expert agent which all are open source. Each of them should be smaller model but specialized in degree.

For example Auto creates an Agent designed to have 10% expertise and 90% general knowledge for brain storming. This Agent can create another Agent with a focus on finance, with 80% Truce but 20% general knowledge. The Finance Agent creates a Model that doesn't exist yet to solve Math and Statistic problems that doesn't exits.

These agents would communicate with each other to train and improve themselves, leveraging collective knowledge and expertise to continuously enhance their capabilities.

I would really love to have these agents in the hand of the open source community.

[zachary-kaelan]

There's a ton of smaller ones that can run relatively efficiently. Glance the ones the issue author noted.

GPT4All | LLaMA

LLaMA requires 14 GB of GPU memory for the model weights on the smallest, 7B model, and with default parameters, it requires an additional 17 GB for the decoding cache (I don't know if that's necessary).

But GPT4All called me out big time with their demo being them chatting about the smallest model's memory requirement of 4 GB. I've never heard of machine learning using 4-bit parameters before, but the math checks out. You'd have to feed it something like this to verify its usability. The full, better performance model on GPU requires 16 GB RAM.

My biggest concern would be the context window size. Both of those look like they're limited to 2048 tokens. The full example AutoGPT prompt is a third of that.

[Neronjust2017]

There's a ton of smaller ones that can run relatively efficiently. Glance the ones the issue author noted. GPT4All | LLaMA

LLaMA requires 14 GB of GPU memory for the model weights on the smallest, 7B model, and with default parameters, it requires an additional 17 GB for the decoding cache (I don't know if that's necessary).

But GPT4All called me out big time with their demo being them chatting about the smallest model's memory requirement of 4 GB. I've never heard of machine learning using 4-bit parameters before, but the math checks out. You'd have to feed it something like this to verify its usability. The full, better performance model on GPU requires 16 GB RAM.

My biggest concern would be the context window size. Both of those look like they're limited to 2048 tokens. The full example AutoGPT prompt is a third of that.

yeah. The number of tokens of questions generated by AutoGPT during the thinking process are quite a lot. Besides, one thing I' m worried about is that the chat models provided by OpenAI, such as gpt-3.5-turbo, take a series of messages as input, as you can find in https://platform.openai.com/docs/guides/chat/introduction. For example, an API call of openai.ChatCompletion.create looks as follows:

# Note: you need to be using OpenAI Python v0.27.0 for the code below to work
import openai

openai.ChatCompletion.create(
  model="gpt-3.5-turbo",
  messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the world series in 2020?"},
        {"role": "assistant", "content": "The Los Angeles Dodgers won the World Series in 2020."},
        {"role": "user", "content": "Where was it played?"}
    ]
)

Messages must be an array of message objects, where each object has a role (either "system", "user", or "assistant") and content (the content of the message). The system, user and assistant messages and the conversation history helps build contextual information for user's final question. However, I have reviewed the source code of https://github.com/nomic-ai/pyllamacpp (offical pythonic API for gpt4all model) and its usage. It seems no similar input argments for gpt4all, which means the conversation history, roles and context information is missing when chatting with gpt4all model, I guess.

[djmaze]

If you want a conversational model, you should probably use Vicuna (based on llama). It supports the human and assistant roles (via string prefixes).

Also, with llama.cpp (or PyLLaMACpp), the memory usage is really low because the models are quantized to 4 bit. Vicuna 13b needs about 5 GB on my machine.

[Neronjust2017]

If you want a conversational model, you should probably use Vicuna (based on llama). It supports the human and assistant roles (via string prefixes).

Also, with llama.cpp (or PyLLaMACpp), the memory usage is really low because the models are quantized to 4 bit. Vicuna 13b needs about 5 GB on my machine.

thanks! One question, why does Vicuna 13b need only 5GB on your machine? It says 7b needs around 30 GB of CPU RAM, and 13b needs around 60 GB of CPU RAM, https://github.com/lm-sys/FastChat. Did you use --low-cpu-mem?

[djmaze]

You need to use llama.cpp (CPU-based) instead of FastChat (GPU-based). FastChat (the original) is more accurate because it operates in floating point, but it also needs much more RAM. Plus some additional tweaks that llama.cpp makes.

[Neronjust2017]

You need to use llama.cpp (CPU-based) instead of FastChat (GPU-based). FastChat (the original) is more accurate because it operates in floating point, but it also needs much more RAM. Plus some additional tweaks that llama.cpp makes.

So what I should do is to obtain original LLaMA model weights (also the Vicuna delta wights) and place theme in right place, convert the 7B or 13B model to ggml FP16 or INT4 format, and finally run the inference, right? In INT4 quantized size, memory size of 7B model reduces to 3.9GB. Also, FastChat provides a similar API with OpenAI's, for example:

import os
from fastchat import client

client.set_baseurl(os.getenv("FASTCHAT_BASEURL"))

completion = client.ChatCompletion.create(
  model="vicuna-7b-v1.1",
  messages=[
    {"role": "user", "content": "Hello!"}
  ]
)

print(completion.choices[0].message)

can I specify roles, such as "user", "assistant" in the command line using llama.cpp? can I use fastchat.client.ChatCompletion. create to infer the quantified model? Besides, does fastchat provide API to create embeddings, just likeopenai.Embedding.create? does they provide open source embedding models like text-embedding-ada-002?

[keldenl]

i posted this in another related thread but i got autogpt mostly working a couple times but embeddings seems to be the wall im hitting (hardcoded different embedding sizes to make it work)

https://github.com/keldenl/gpt-llama.cpp aims to replace openais api endpoints completely with llama.cpp (including embeddings). but i don't know much about embeddings and seems to work off and on, anybody got any pointers there?

[arronKler]

just replace the request to openai with your own models service in llm_utils.py. But the embedding part may need to keep using Openai's embedding api.

Can I avoid using OpenAI's embedding APIs? Because the network connection to OpenAI can not be always established successfully, I want to be totally offline.

of course, the embedding API call is used for weaviate and milvus, if you dont need these two memory storage, you can just do nothing for the embedding part. replace llm_utils is enough @Neronjust2017

[djmaze]

@Neronjust2017

So what I should do is to obtain original LLaMA model weights (also the Vicuna delta wights) and place theme in right place, convert the 7B or 13B model to ggml FP16 or INT4 format, and finally run the inference, right?

You can get the full ggml at huggingface.

can I specify roles, such as "user", "assistant" in the command line using llama.cpp?

I don't think so. You need to use the roles inside the prompt text as described in Vicuna's documentation.

About the other FastChat questions: don't know, I only used llama.cpp so far.

[keldenl]

i posted this in another related thread but i got autogpt mostly working a couple times but embeddings seems to be the wall im hitting (hardcoded different embedding sizes to make it work)

https://github.com/keldenl/gpt-llama.cpp aims to replace openais api endpoints completely with llama.cpp (including embeddings). but i don't know much about embeddings and seems to work off and on, anybody got any pointers there?

i got autogpt working with llama.cpp! see https://github.com/keldenl/gpt-llama.cpp/issues/2#issuecomment-1514353829

i'm using vicuna for embeddings and generation but it's struggling a bit to generate proper commands to not fall into a infinite loop of attempting to fix itself X( will look into this tmr but super exciting cuz i got the embeddings working! (turns out it was a bug on my end lol)

here's a screenshot 🎉

edit: had to make some changes to autogpt (add base_url to openai_base_url, and adjust the dimensions of the vector, but otherwise left it alone)

[Sorann753]

the https://github.com/oobabooga/text-generation-webui is able to run in an API mode if you use the flags --listen --no-stream

maybe we could make an option to allow users to use their local webui as a server instead of the OpenAI API? something like adding USE_LOCAL_SERVER=True to the .env and then in a local_llm.yaml file we'd do the configurations needed

Since they are making an interface for running all open source LLM and our project is instead to use a LLM to run an agent in autonomy, it might be a good idea to avoid redoing what they already did

[Neronjust2017]

i posted this in another related thread but i got autogpt mostly working a couple times but embeddings seems to be the wall im hitting (hardcoded different embedding sizes to make it work) https://github.com/keldenl/gpt-llama.cpp aims to replace openais api endpoints completely with llama.cpp (including embeddings). but i don't know much about embeddings and seems to work off and on, anybody got any pointers there?

i got autogpt working with llama.cpp! see keldenl/gpt-llama.cpp#2 (comment)

i'm using vicuna for embeddings and generation but it's struggling a bit to generate proper commands to not fall into a infinite loop of attempting to fix itself X( will look into this tmr but super exciting cuz i got the embeddings working! (turns out it was a bug on my end lol)

here's a screenshot tada

edit: had to make some changes to autogpt (add base_url to openai_base_url, and adjust the dimensions of the vector, but otherwise left it alone)

nice work! what local embedding model did you test?

[keldenl]

i just utilized the embeddings.cpp example from llama.cpp with llama-based models!

[DGdev91]

Thanks to @keldenl for his work!

I made a pull request for the changes mentioned by him #2594

[keldenl]

beautiful work @DGdev91 ! that does the trick. i'll go ahead and write an extensive guide tonight and link it here as well

[keldenl]

✨ FULL GUIDE POSTED on how to get Auto-GPT running with llama.cpp via gpt-llama.cpp in https://github.com/keldenl/gpt-llama.cpp/issues/2#issuecomment-1515738173.

Huge shoutout to @DGdev91 for the PR and hope it gets merged soon!

[mudler]

Hi 👋

I'm the author of https://github.com/go-skynet/LocalAI, I'd be glad to help out in what's missing to see this working with local models - LocalAI has multiple backends, is multi-models and keeps thing in memory for faster inference. Supports models including gpt4all-j and those supported by llama.cpp, and will also have support for Cerebras.

Happy to jump in!

[DGdev91]

Hi 👋

I'm the author of https://github.com/go-skynet/LocalAI, I'd be glad to help out in what's missing to see this working with local models - LocalAI has multiple backends, is multi-models and keeps thing in memory for faster inference. Supports models including gpt4all-j and those supported by llama.cpp, and will also have support for Cerebras.

Happy to jump in!

Cool! As long it follows OpenAI's API, it should already be possible to use it with my changes

[mudler]

Hi 👋 I'm the author of https://github.com/go-skynet/LocalAI, I'd be glad to help out in what's missing to see this working with local models - LocalAI has multiple backends, is multi-models and keeps thing in memory for faster inference. Supports models including gpt4all-j and those supported by llama.cpp, and will also have support for Cerebras. Happy to jump in!

Cool! As long it follows OpenAI's API, it should already be possible to use it with my changes

Super, thanks @DGdev91 🙏!

[DGdev91]

@mudler i tried out your project. i can confirm that is already possible to use it with my changes. just set OPENAI_API_BASE_URL=http://localhost:8080/v1 and EMBED_DIM= depending on the model you are using (4096 for 7B, 5120 for 13B, 6656 for 33B, 8192 for 65B)

There's just a minor issue: AutoGPT expects gpt-3.5-turbo or gpt-4 as the model IDs, while you are using the model file name as id. A quick (and hacky) fix in your case could be renaming the model in the "models" folder as "gpt-3.5-turbo", that worked in my case. It could also be a good idea to make some changes to LocalAI to make it always serve the first model he finds (or one wich has been configured for that) as gpt-3.5-turbo and gpt4. after all, the idea here is to offer a drop-in replacement for those models.

Also, i don't know why, but it seems much slower than keldenl's implementation. Still kinda works, but isn't really practical to use.

Finally, remember that some LLMs work better than others for this. the first tests have been made with Vicuna13B. I tried also GPT4All, but it can't create a valid json as response