mem0

mem0 copied to clipboard

🚀 The feature

LM Studio (https://lmstudio.ai) allows for easy downloading and running of model off of Huggingface locally. While it ostensibly is supporting the OpenAI API, when I use Embedchain with the OpenAI setting for the LLM in Embedchain (https://docs.embedchain.ai/components/llms#openai) pointed to a local URL, I see this error in the LM Studio console:

llm:
  provider: openai
  config:
    endpoint: http://localhost:1234/v1/chat/completions

or

llm:
  provider: huggingface
  config:
    endpoint: http://localhost:1234/v1/chat/completions

Results in:

Error asking LLM -> Error raised by inference API: 'messages' field is required

LM Studio is expecting this:

curl http://localhost:1234/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{ 
  "messages": [ 
    { "role": "system", "content": "Always answer in rhymes." },
    { "role": "user", "content": "Introduce yourself." }
  ], 
  "temperature": 0.7, 
  "max_tokens": -1,
  "stream": false
}'

I will poke around with this and see if I may send a pull request by creating an lm_studio.py:

https://github.com/jsgoecke/embedchain/blob/main/embedchain/llm/openai.py

Motivation, pitch

LM Studio let's me host models off of Huggingface locally, as many models I run on Huggingface don't have the pipeline_tag "text-generation" to let me launch there via Embedchain.

this is very important.

I would like support for LMStudio, Ollama is limited

I too wanted to make the two work together and finally found a way.

Just use the OpenAI provider and set environment variable to use localhost and it works out of the box:

import os
from embedchain import App

os.environ["OPENAI_API_BASE"] = "http://localhost:1234/v1"
os.environ["OPENAI_API_KEY"] = "lm-studio"

config = {
  'app': {
   'config': {
      'name': 'RAG 1',
      'collect_metrics': False,
      'id': 'rag-1'
    }
  },
  'llm': {
    'provider': 'openai',
    'config': {
      'model': 'lmstudio-community/Meta-Llama-3-8B-Instruct-GGUF',
      'temperature': 0.7,
      'number_documents': 1,
      'stream': True
    }
  },
  'embedder': {
    'provider': 'openai',
    'config': {
      'model': 'nomic-ai/nomic-embed-text-v1.5-GGUF',
      'vector_dimension': 1536
    }
  },
  'vectordb': {
    'provider': 'chroma',
    'config': {
      'collection_name': 'rag-1',
      'dir': 'db',
      'allow_reset': True
    }
  }
}

app = App.from_config(config=config)

It would be nice to put this in the documentation :-)