Is there a way to use a local embedding model?

[firdausai]

From my testing, embedding and indexing new paper seems to be the one eating my rate limit and quota, so I am looking to use a locally hosted embedding model. The docs said that paper-qa accepts any embedding model name supported by litellm, but I don't see a section for using locally hosted embedding model.

[dosubot[bot]]

Hey @firdausai! I'm here to help you with any bugs, questions, or contributions. Let's tackle this together!

Yes, you can use a local embedding model with paper-qa. Here is an example of how to change the embedding model:

from paperqa import Settings, ask

answer = ask(
    "What manufacturing challenges are unique to bispecific antibodies?",
    settings=Settings(embedding="text-embedding-3-large"),
)

Additionally, you can customize hybrid embeddings:

from paperqa import (
    Docs,
    HybridEmbeddingModel,
    SparseEmbeddingModel,
    LiteLLMEmbeddingModel,
)

model = HybridEmbeddingModel(
    models=[LiteLLMEmbeddingModel(), SparseEmbeddingModel(ndim=1024)]
)
docs = Docs()
for doc in ("myfile.pdf", "myotherfile.pdf"):
    docs.add(doc, embedding_model=model)

This allows you to specify and use different embedding models supported by LiteLLM. The HybridEmbeddingModel combines multiple embedding models, such as LiteLLMEmbeddingModel and SparseEmbeddingModel, to enhance the embedding process.

For more advanced use cases, you can also define custom LLM models and use them with paper-qa:

from paperqa.llms import LLMModel, Chunk
from typing import AsyncIterable

class MyLLM(LLMModel):
    name: str = "myllm"

    async def acomplete(self, prompt: str) -> Chunk:
        return Chunk(text="Echo", prompt_tokens=1, completion_tokens=1)

    async def acomplete_iter(self, prompt: str) -> AsyncIterable[Chunk]:
        yield Chunk(text="Echo", prompt_tokens=1, completion_tokens=1)

docs = Docs()
docs.add(
    "myfile.pdf",
    citation="My Citation, 2023, Accessed now",
    dockey="test",
    llm_model=MyLLM(),
)
evidence = docs.get_evidence("Echo", summary_llm_model=MyLLM()).contexts
assert "Echo" in evidence[0].context

This example demonstrates how to create a custom LLM model and integrate it with paper-qa for document processing and evidence extraction ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[firdausai]

How about using ollama as an embedding provider, is it possible? @dosu

[dosubot[bot]]

I couldn't find any specific mention of Ollama as an embedding provider in the paper-qa codebase or documentation ^[1][2]. However, you can integrate a new embedding provider by following these steps:

1. Implement the EmbeddingModel class:

   from paperqa.llms import EmbeddingModel
   
   class OllamaEmbeddingModel(EmbeddingModel):
       name: str = "ollama"
   
       async def embed_documents(self, texts: list[str]) -> list[list[float]]:
           # Your custom embedding logic for Ollama here
           embeddings = []
           for text in texts:
               # Example: Convert each text to a list of floats (dummy implementation)
               embedding = [float(ord(char)) for char in text]
               embeddings.append(embedding)
           return embeddings
   

2. Use the custom embedding model when adding documents:

   from paperqa import Docs
   
   docs = Docs()
   for doc in ("myfile.pdf", "myotherfile.pdf"):
       docs.add(doc, embedding_model=OllamaEmbeddingModel())
   

3. Alternatively, specify the embedding model in the Settings object:

   from paperqa import Settings, ask
   
   settings = Settings(embedding="ollama")
   answer = ask(
       "What manufacturing challenges are unique to bispecific antibodies?",
       settings=settings,
   )
   

By implementing the EmbeddingModel class and using it in your Docs or Settings, you can integrate Ollama or any other custom embedding provider into paper-qa ^[3][1].

To continue talking to Dosu, mention @dosu.