ragas
ragas copied to clipboard
explodinggradients
TestsetGenerator in local Ollama
[ ] I have checked the documentation and related resources and couldn't resolve my bug.
Describe the bug I am not able to do testset generation using the local Ollama models.
Ragas version: 0.2.16.dev2+gb284337 Python version: 3.11.5
Code to Reproduce
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper
generator_llm = LangchainLLMWrapper(Ollama(model='gemma3:27',request_timeout=120.0,temperature=0))
generator_embeddings = LangchainEmbeddingsWrapper(HuggingFaceEmbedding(model_name="mixedbread-ai/mxbai-embed-large-v1",truncate_dim=512,local_files_only=False,embed_batch_size=8,trust_remote_code=True))
from ragas.testset import TestsetGenerator
generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
dataset = generator.generate_with_langchain_docs(docs, testset_size=10)
return dataset
Error trace
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[17], [line 1](vscode-notebook-cell:?execution_count=17&line=1)
----> [1](vscode-notebook-cell:?execution_count=17&line=1) dataset=langchain_local(docs)
File ~/llm/AI-Assistant/johndoe/../src/eval/ragas_eval.py:50, in langchain_local(docs)
[47](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/src/eval/ragas_eval.py:47) from ragas.testset import TestsetGenerator
[49](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/src/eval/ragas_eval.py:49) generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
---> [50](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/src/eval/ragas_eval.py:50) dataset = generator.generate_with_langchain_docs(docs, testset_size=10)
[51](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/src/eval/ragas_eval.py:51) return dataset
File ~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:188, in TestsetGenerator.generate_with_langchain_docs(self, documents, testset_size, transforms, transforms_llm, transforms_embedding_model, query_distribution, run_config, callbacks, with_debugging_logs, raise_exceptions)
[185](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:185) apply_transforms(kg, transforms)
[186](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:186) self.knowledge_graph = kg
--> [188](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:188) return self.generate(
[189](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:189) testset_size=testset_size,
[190](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:190) query_distribution=query_distribution,
[191](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:191) run_config=run_config,
[192](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:192) callbacks=callbacks,
[193](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:193) with_debugging_logs=with_debugging_logs,
[194](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:194) raise_exceptions=raise_exceptions,
[195](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:195) )
File ~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:369, in TestsetGenerator.generate(self, testset_size, query_distribution, num_personas, run_config, batch_size, callbacks, token_usage_parser, with_debugging_logs, raise_exceptions)
[366](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:366) patch_logger("ragas.experimental.testset.transforms", logging.DEBUG)
[368](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:368) if self.persona_list is None:
--> [369](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:369) self.persona_list = generate_personas_from_kg(
[370](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:370) llm=self.llm,
[371](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:371) kg=self.knowledge_graph,
[372](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:372) num_personas=num_personas,
[373](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:373) callbacks=callbacks,
[374](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:374) )
[375](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:375) else:
[376](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/synthesizers/generate.py:376) random.shuffle(self.persona_list)
File ~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/persona.py:95, in generate_personas_from_kg(kg, llm, persona_generation_prompt, num_personas, filter_fn, callbacks)
[93](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/persona.py:93) nodes = [node for node in kg.nodes if filter_fn(node)]
[94](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/persona.py:94) if len(nodes) == 0:
---> [95](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/persona.py:95) raise ValueError(
[96](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/persona.py:96) "No nodes that satisfied the given filer. Try changing the filter."
[97](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/persona.py:97) )
[99](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/persona.py:99) summaries = [node.properties.get("summary") for node in nodes]
[100](https://vscode-remote+ssh-002dremote-002b192-002e168-002e1-002e75.vscode-resource.vscode-cdn.net/home/_johndoe/llm/AI-Assistant/johndoe/~/llm/AI-Assistant/.venv/lib/python3.11/site-packages/ragas/testset/persona.py:100) summaries = [summary for summary in summaries if isinstance(summary, str)]
ValueError: No nodes that satisfied the given filer. Try changing the filter.
Expected behavior TestsetGenerator similar to the OpenAI call.
Additional context
- In this case, I have tried with Langchain and Ollama, but my final goal would be to LLamaindex and Ollama instead. But wrt to the original tutorial, just wanted to change one variable (OpenAI to Ollama) only first.
- Also checked chatgpt
Replace Ollama with a LangChain LLM that supports agenerate_prompt, like ChatOpenAI
Any ideas if ragas has any plans to support a locall llm, as there is restriction for our project to use cloud/web based llms.
Just to add, not sure, if it is a bug. Mod feel free to change the label, if required.