ragas
ragas copied to clipboard
explodinggradients
Tried Generation Test Set from Together APIs and Hugging Face Embeddings
[ ] I have checked the documentation and related resources and couldn't resolve my bug.
Describe the bug Tried Generation Test Set from Together APIs and Hugging Face Embeddings
Ragas version:0.1.7 Python version: 3.9
Code to Reproduce
import os
from langchain_community.document_loaders import PyPDFDirectoryLoader
from langchain.embeddings import SentenceTransformerEmbeddings
from ragas.testset.generator import TestsetGenerator
from langchain_together import Together
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
generator_llm = Together(
model="mistralai/Mistral-7B-Instruct-v0.2",
together_api_key=os.getenv("TOGETHER_API_KEY")
)
critic_llm = Together(
model="meta-llama/Llama-3-70b-chat-hf",
together_api_key=os.getenv("TOGETHER_API_KEY")
)
embeddings = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
generator = TestsetGenerator.from_langchain(
generator_llm,
critic_llm,
embeddings
)
loader = PyPDFDirectoryLoader("FilePath")
documents = loader.load()
# generate testset
testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
print(testset)
testset.to_pandas().to_csv()
Error trace
Expected behavior
Filename and doc_id are the same for all nodes.
Generating: 0%| | 0/10 [00:00<?, ?it/s]Failed to parse output. Returning None.
Generating: 0%| | 0/10 [00:10<?, ?it/s]
Exception in thread Thread-5:
Traceback (most recent call last):
File "/home//anaconda3/envs/rag-env/lib/python3.9/threading.py", line 980, in _bootstrap_inner
self.run()
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 96, in run
results = self.loop.run_until_complete(self._aresults())
File "/home//anaconda3/envs/rag-env/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
return future.result()
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 84, in _aresults
raise e
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 79, in _aresults
r = await future
File "/home//anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py", line 611, in _wait_for_one
return f.result() # May raise f.exception().
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 38, in sema_coro
return await coro
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
return counter, await callable(*args, **kwargs)
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 142, in evolve
) = await self._aevolve(current_tries, current_nodes)
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 290, in _aevolve
passed = await self.node_filter.filter(merged_node)
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/filters.py", line 60, in filter
output["score"] = sum(output.values()) / len(output.values())
ZeroDivisionError: division by zero
Traceback (most recent call last):
File "/home//Documents/RAG Pipeline/new_demo/test.py", line 34, in <module>
testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/generator.py", line 179, in generate_with_langchain_docs
return self.generate(
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/generator.py", line 274, in generate
raise ExceptionInRunner()
ragas.exceptions.ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
Exception ignored in: <coroutine object Executor.wrap_callable_with_index.<locals>.wrapped_callable_async at 0x7f2d728700c0>
Traceback (most recent call last):
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 142, in evolve
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 545, in _aevolve
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 373, in _acomplex_evolution
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 290, in _aevolve
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/filters.py", line 58, in filter
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/llms/output_parser.py", line 67, in aparse
KeyError: 'idle_for'
Exception ignored in: <coroutine object Executor.wrap_callable_with_index.<locals>.wrapped_callable_async at 0x7f2d728709c0>
Traceback (most recent call last):
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 142, in evolve
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 460, in _aevolve
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 290, in _aevolve
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/filters.py", line 58, in filter
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/llms/output_parser.py", line 67, in aparse
KeyError: 'idle_for'
Exception ignored in: <coroutine object Executor.wrap_callable_with_index.<locals>.wrapped_callable_async at 0x7f2d72870bc0>
Traceback (most recent call last):
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 142, in evolve
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 545, in _aevolve
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 373, in _acomplex_evolution
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 306, in _aevolve
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/filters.py", line 89, in filter
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/llms/base.py", line 92, in generate
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/_asyncio.py", line 142, in async_wrapped
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/_asyncio.py", line 58, in __call__
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/_asyncio.py", line 110, in iter
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/_asyncio.py", line 78, in inner
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/__init__.py", line 420, in next_action
KeyError: 'idle_for'
Exception ignored in: <coroutine object Executor.wrap_callable_with_index.<locals>.wrapped_callable_async at 0x7f2d729a4cc0>
Traceback (most recent call last):
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 142, in evolve
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 298, in _aevolve
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/llms/base.py", line 92, in generate
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/_asyncio.py", line 142, in async_wrapped
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/_asyncio.py", line 58, in __call__
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/_asyncio.py", line 110, in iter
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/_asyncio.py", line 78, in inner
File "/home//anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/__init__.py", line 420, in next_action
KeyError: 'idle_for'
Task was destroyed but it is pending!
task: <Task pending name='Task-677' coro=<as_completed.<locals>.sema_coro() running at /home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d728b7160>()]> cb=[as_completed.<locals>._on_completion() at /home//anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Task was destroyed but it is pending!
task: <Task pending name='Task-675' coro=<as_completed.<locals>.sema_coro() running at /home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d72910ca0>()]> cb=[as_completed.<locals>._on_completion() at /home//anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Task was destroyed but it is pending!
task: <Task pending name='Task-673' coro=<as_completed.<locals>.sema_coro() running at /home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d728b70a0>()]> cb=[as_completed.<locals>._on_completion() at /home/'/anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Task was destroyed but it is pending!
task: <Task pending name='Task-681' coro=<as_completed.<locals>.sema_coro() running at /home/'/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d72970b50>()]> cb=[as_completed.<locals>._on_completion() at /home//anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Task was destroyed but it is pending!
task: <Task pending name='Task-674' coro=<as_completed.<locals>.sema_coro() running at /home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d726395e0>()]> cb=[as_completed.<locals>._on_completion() at /home//anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Task was destroyed but it is pending!
task: <Task pending name='Task-678' coro=<as_completed.<locals>.sema_coro() running at /home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d72903670>()]> cb=[as_completed.<locals>._on_completion() at /home//anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Task was destroyed but it is pending!
task: <Task pending name='Task-676' coro=<as_completed.<locals>.sema_coro() running at /home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d726352b0>()]> cb=[as_completed.<locals>._on_completion() at /home//anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Task was destroyed but it is pending!
task: <Task pending name='Task-679' coro=<as_completed.<locals>.sema_coro() running at /home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d727dad90>()]> cb=[as_completed.<locals>._on_completion() at /home//anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Task was destroyed but it is pending!
task: <Task pending name='Task-682' coro=<as_completed.<locals>.sema_coro() running at /home//anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d72957640>()]> cb=[as_completed.<locals>._on_completion() at /home//anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Additional context Add any other context about the problem here.
Have a look at some changes that allowed me to use together ai:
https://github.com/explodinggradients/ragas/pull/936
hey @Eknathabhiram, this is a class of bugs that are due to how we interface with different LLM providers. Doing a broader fix for this soon 🙂
hey @ciekawy thank you so much for putting a fix for this and my sinciear apologies for not taking a closer look at this earlier 🙏🏽 will take a look at this today and leave a review and get it merged soon 🙂 ❤️
