explodinggradients
[R-313] Adding more LLM/Embedding Providers
A running issue to get feedback on which LLM or Embedding Providers we need to add for you
If the output of my model is obtained through a method call, for example, def query_model(query): return response. Then how should I write the code to load the model? In addition, the Embedding model is also generated through a similar method call, providing methods for converting words into vectors and comparing vector similarities.
hey @kent-william007, in that case you will have to implement https://github.com/explodinggradients/ragas/blob/1e7121be03b5ed5a3ee24bfa4aaf5a875337068d/src/ragas/llms/base.py#L64-L83
on your own and make sure make sure that the result is of the type https://python.langchain.com/api_reference/core/outputs/langchain_core.outputs.llm_result.LLMResult.html
similarly for https://github.com/explodinggradients/ragas/blob/main/src/ragas/embeddings/base.py
let me know if you have further doubts, we'll help you out 🙂
btw are you on discord?
Is it possible to support model invocation through an API? This way, we can leverage the model capabilities provided by the server without needing to worry about which underlying model is being used.
Trying to use Ragas with Google's Gemini models always gives me some headache.
Whether I'm Evaluating Using Metrics or trying to Generate Synthetic Testset for RAG I keep getting errors such as The LLM generation was not completed. Please increase try increasing the max_tokens and try again.
Same result with different models e.g. gemini-1.5-pro-001,gemini-1.5-pro-002, gemini-1.5-flash.
I just commented on #1573 about getting the same thing.
Gemini support, please. 🟦🟥🟨🟩
@matheusft could you check this PR and docs: https://github.com/explodinggradients/ragas/pull/1728 this should help you.
let me know if it solves your issue or I'll look further into it
@jjmachan I was having the same issue using google`s api. Seens like the #1728 works here with the googles api.
@matheusft could you check this PR and docs: #1728 this should help you.
let me know if it solves your issue or I'll look further into it
It does not work for me as LangchainLLMWrapper does not implement the .invoke() method, so it's not working with my LangChain RAG.
I'm now getting
Expected a Runnable, callable or dict.Instead got an unsupported type: <class 'ragas.llms.base.LangchainLLMWrapper'>
i get this error
AttributeError: 'GoogleGenerativeAIEmbeddings' object has no attribute 'set_run_config'
I used the LangchainLLMWrapper for the llm, but there seems to be an issue with embeddings.
I don't know why most libraries strictly enforces OPENAI when not all uses Open AI
Note: I am using gemini llm and embeddings
@matheusft could you check this PR and docs: #1728 this should help you. let me know if it solves your issue or I'll look further into it
It does not work for me as
LangchainLLMWrapperdoes not implement the.invoke()method, so it's not working with my LangChain RAG.I'm now getting
Expected a Runnable, callable or dict.Instead got an unsupported type: <class 'ragas.llms.base.LangchainLLMWrapper'>
wrapped classes are only intended to be used inside Ragas, it is recommended to not use it anywhere else
Trying to use Ragas with Google's Gemini models always gives me some headache.
Whether I'm Evaluating Using Metrics or trying to Generate Synthetic Testset for RAG I keep getting errors such as
The LLM generation was not completed. Please increase try increasing the max_tokens and try again.Same result with different models e.g.
gemini-1.5-pro-001,gemini-1.5-pro-002,gemini-1.5-flash.I just commented on #1573 about getting the same thing.
Gemini support, please. 🟦🟥🟨🟩
So would you have any suggestion on how to use Ragas with Gemini and Langchain that actually works?
hey @matheusft let me check this and get back to you. We had a community PR that should have helped fix this but I think that didn't solve the issue your facing? let me setup the google gemini workflow and get back to you by today 🙂
Does anyone know how else on can evaluate a rag system without ragas?
hey @matheusft could you check this PR out: https://github.com/explodinggradients/ragas/pull/1759
I've tested it on my end with gemini but do give it a go? I'll merge it after that 🙂
hey @matheusft could you check this PR out: #1759
I've tested it on my end with gemini but do give it a go? I'll merge it after that 🙂
Fantastic! I’ll give it a go this week and will let you know the outcome.
Thanks in advance.
Hey @matheusft could you check this PR out: #1759
I've tested it on my end with gemini but do give it a go? I'll merge it after that 🙂
Hey,
I just finished testing it.
The latest version (0.2.7) is working fine for running evaluate (this code) using Gemini, no warnings or errors. 👏🏻
However, still getting ERROR unable to apply transformation: The LLM generation was not completed. Please increase try increasing the max_tokens and try again. when trying to run Generate Testset code with Gemini.
from langchain_google_vertexai import VertexAI
from langchain_google_vertexai import VertexAIEmbeddings
from ragas.embeddings import LangchainEmbeddingsWrapper
from ragas.llms import LangchainLLMWrapper
from ragas.testset import TestsetGenerator
llm = VertexAI(model_name="MY_LLM_MODEL")
embedding_function = VertexAIEmbeddings(
model_name="MY_EMBEDDING_MODEL",
project_id = "MY_PROJECT",
location = "MY_LOCATION",
request_parallelism=1,
)
evaluator_llm = LangchainLLMWrapper(llm)
evaluator_embeddings = LangchainEmbeddingsWrapper(embedding_function)
generator = TestsetGenerator(llm=evaluator_llm, embedding_model=evaluator_embeddings)
dataset = generator.generate_with_langchain_docs(docs, testset_size=10)
Hey @matheusft could you check this PR out: #1759 I've tested it on my end with gemini but do give it a go? I'll merge it after that 🙂
Hey,
I just finished testing it. The latest version (
0.2.7) is working fine for runningevaluate(this code) usingGemini, no warnings or errors. 👏🏻However, still getting
ERROR unable to apply transformation: The LLM generation was not completed. Please increase try increasing the max_tokens and try again.when trying to run Generate Testset code withGemini.from langchain_google_vertexai import VertexAI from langchain_google_vertexai import VertexAIEmbeddings from ragas.embeddings import LangchainEmbeddingsWrapper from ragas.llms import LangchainLLMWrapper from ragas.testset import TestsetGenerator llm = VertexAI(model_name="MY_LLM_MODEL") embedding_function = VertexAIEmbeddings( model_name="MY_EMBEDDING_MODEL", project_id = "MY_PROJECT", location = "MY_LOCATION", request_parallelism=1, ) evaluator_llm = LangchainLLMWrapper(llm) evaluator_embeddings = LangchainEmbeddingsWrapper(embedding_function) generator = TestsetGenerator(llm=evaluator_llm, embedding_model=evaluator_embeddings) dataset = generator.generate_with_langchain_docs(docs, testset_size=10)
Hey @matheusft, are Ragas metrics available for gemini model without using langchain?
hey @kailashp19 not right now sadly - you have to use either langchain or llamaindex
how do you use the API right now?
@matheusft really sorry about the late response but were you able to figure it out? seeemed like out of context error - we did some improvements there but could be a model configuration issue too?
Hello, no worries. No updates on my side, apologies. I've moved as as I could not get past this error.
hey @kailashp19 not right now sadly - you have to use either langchain or llamaindex
how do you use the API right now?
@matheusft really sorry about the late response but were you able to figure it out? seeemed like out of context error - we did some improvements there but could be a model configuration issue too?
Actually, our team is trying to use RAGAs metrics with other LLMs like gemini, groq api. However it is not supporting with other LLMs. Any plans when RAGAs metrics will be usable for other LLMs?
@kailashp19 we do support other LLMs as well via Langchain and Llamaindex, do checkout this notebook doc https://docs.ragas.io/en/stable/howtos/customizations/customize_models/ ?
let me know if you're still facing trouble
@matheusft can I suggest maybe coming for an office hour? we would love to help you setup evals and get you unblocked if you give us a chance again 🙂 (link https://cal.com/team/ragas/office-hours)
Does anyone have any idea if I can use free hugging face models for RAGAS metrics?
