ragas
ragas copied to clipboard
explodinggradients
TestsetGenerator -> RuntimeError: ... got Future <..> attached to a differen t loop
This is related to: https://github.com/explodinggradients/ragas/issues/681
Describe the bug TestGeneration launches an error with default example of https://docs.ragas.io/en/stable/concepts/testset_generation.html with VertexAI models.
Ragas version: 0.1.7 Python version: Python 3.10.13
Code to Reproduce
generator_llm = VertexAI(model_name="text-bison", temperature=0, response_validation=False)
critic_llm = VertexAI(model_name="gemini-pro", temperature=0, response_validation=False)
vertexai_embeddings = VertexAIEmbeddings("textembedding-gecko@001")
generator = TestsetGenerator.from_langchain(
generator_llm,
critic_llm,
vertexai_embeddings
)
distributions = {
simple: 0.5,
multi_context: 0.4,
reasoning: 0.1
}
loader = PubMedLoader("liver", load_max_docs=10)
documents = loader.load()
testset = generator.generate_with_langchain_docs(documents, 10, distributions, with_debugging_logs=True)
testset.to_pandas()
Error trace
Filename and doc_id are the same for all nodes.
Generating: 0%| | 0[/10](http://localhost:8889/10) [00:00<?, ?it[/s](http://localhost:8889/s)][ragas.testset.filters.DEBUG] context scoring: {'clarity': 3, 'depth': 3, 'structure': 3, 'relevance': 3, 'score': 3.0}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Hyperuricemia', 'Protein peptides', 'Anti-hyperuricemia properties', 'Uric acid-lowering peptides', 'Functional foods']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 3, 'depth': 3, 'structure': 3, 'relevance': 3, 'score': 3.0}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Liver surgery', 'Liver anatomy and physiology', 'Liver resection', 'Vascular control', 'Liver transplantation', 'Residual liver function']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 2, 'depth': 2, 'structure': 2, 'relevance': 3, 'score': 2.25}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Ammonia sensor', 'Colorimetric optical readout', 'Plasmonic Ag[/SiO](http://localhost:8889/SiO) nanoparticles', 'Animal serum', 'Paper-based sensor']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 1, 'depth': 1, 'structure': 1, 'relevance': 1, 'score': 1.0}
[ragas.testset.evolutions.INFO] retrying evolution: 0 times
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 3, 'depth': 3, 'structure': 3, 'relevance': 3, 'score': 3.0}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Liver surgery', 'Liver anatomy and physiology', 'Liver resection', 'Vascular control', 'Liver transplantation', 'Residual liver function']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 3, 'depth': 3, 'structure': 3, 'relevance': 3, 'score': 3.0}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Hyperuricemia', 'Protein peptides', 'Anti-hyperuricemia properties', 'Uric acid-lowering peptides', 'Functional foods']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 3, 'depth': 2, 'structure': 3, 'relevance': 3, 'score': 2.75}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Metastatic breast cancer', 'Age groups', 'Metastatic patterns', 'Survival outcomes', 'Elderly patients']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 3, 'depth': 2, 'structure': 3, 'relevance': 3, 'score': 2.75}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Metastatic breast cancer', 'Age groups', 'Metastatic patterns', 'Survival outcomes', 'Elderly patients']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 2, 'depth': 2, 'structure': 2, 'relevance': 3, 'score': 2.25}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Ammonia sensor', 'Colorimetric optical readout', 'Plasmonic Ag[/SiO](http://localhost:8889/SiO) nanoparticles', 'Animal serum', 'Paper-based sensor']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 3, 'depth': 2, 'structure': 3, 'relevance': 3, 'score': 2.75}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Metastatic breast cancer', 'Age groups', 'Metastatic patterns', 'Survival outcomes', 'Elderly patients']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 3, 'depth': 3, 'structure': 3, 'relevance': 3, 'score': 3.0}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Liver surgery', 'Liver anatomy and physiology', 'Liver resection', 'Vascular control', 'Liver transplantation', 'Residual liver function']
Generating: 0%| | 0[/10](http://localhost:8889/10) [05:20<?, ?it[/s](http://localhost:8889/s)]
Exception in thread Thread-31:
Traceback (most recent call last):
File "/Users/.../miniconda3/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/Users/.../venv/lib/python3.10/site-packages/ragas/executor.py", line 96, in run
results = self.loop.run_until_complete(self._aresults())
File "/Users/.../venv/lib/python3.10/site-packages/nest_asyncio.py", line 98, in run_until_complete
return f.result()
File "/Users/.../miniconda3/lib/python3.10/asyncio/futures.py", line 201, in result
raise self._exception.with_traceback(self._exception_tb)
File "/Users/.../miniconda3/lib/python3.10/asyncio/tasks.py", line 232, in __step
result = coro.send(None)
File "/Users/.../venv/lib/python3.10/site-packages/ragas/executor.py", line 84, in _aresults
raise e
File "/Users/.../venv/lib/python3.10/site-packages/ragas/executor.py", line 79, in _aresults
r = await future
File "/Users/.../miniconda3/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
return f.result() # May raise f.exception().
File "/Users/.../miniconda3/lib/python3.10/asyncio/futures.py", line 201, in result
raise self._exception.with_traceback(self._exception_tb)
File "/Users/.../miniconda3/lib/python3.10/asyncio/tasks.py", line 232, in __step
result = coro.send(None)
File "/Users/.../venv/lib/python3.10/site-packages/ragas/executor.py", line 38, in sema_coro
return await coro
File "/Users/.../venv/lib/python3.10/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
return counter, await callable(*args, **kwargs)
File "/Users/.../venv/lib/python3.10/site-packages/ragas/testset/evolutions.py", line 142, in evolve
) = await self._aevolve(current_tries, current_nodes)
File "/Users/.../venv/lib/python3.10/site-packages/ragas/testset/evolutions.py", line 298, in _aevolve
results = await self.generator_llm.generate(
File "/Users/.../venv/lib/python3.10/site-packages/ragas/llms/base.py", line 92, in generate
return await agenerate_text_with_retry(
File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 142, in async_wrapped
return await fn(*args, **kwargs)
File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 58, in __call__
do = await self.iter(retry_state=retry_state)
File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 110, in iter
result = await action(retry_state)
File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 78, in inner
return fn(*args, **kwargs)
File "/Users/.../venv/lib/python3.10/site-packages/tenacity/__init__.py", line 410, in exc_check
raise retry_exc.reraise()
File "/Users/.../venv/lib/python3.10/site-packages/tenacity/__init__.py", line 183, in reraise
raise self.last_attempt.result()
File "/Users/.../miniconda3/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/Users/.../miniconda3/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 61, in __call__
result = await fn(*args, **kwargs)
File "/Users/.../venv/lib/python3.10/site-packages/ragas/llms/base.py", line 177, in agenerate_text
result = await self.langchain_llm.agenerate_prompt(
File "/Users/.../venv/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 643, in agenerate_prompt
return await self.agenerate(
File "/Users/.../venv/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 1018, in agenerate
output = await self._agenerate_helper(
File "/Users/.../venv/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 882, in _agenerate_helper
raise e
File "/Users/.../venv/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 866, in _agenerate_helper
await self._agenerate(
File "/Users/.../venv/lib/python3.10/site-packages/langchain_google_vertexai/llms.py", line 413, in _agenerate
res = await _acompletion_with_retry(
File "/Users/.../venv/lib/python3.10/site-packages/langchain_google_vertexai/llms.py", line 123, in _acompletion_with_retry
return await _acompletion_with_retry_inner(
File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 142, in async_wrapped
return await fn(*args, **kwargs)
File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 58, in __call__
do = await self.iter(retry_state=retry_state)
File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 110, in iter
result = await action(retry_state)
File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 78, in inner
return fn(*args, **kwargs)
File "/Users/.../venv/lib/python3.10/site-packages/tenacity/__init__.py", line 390, in <lambda>
self._add_action_func(lambda rs: rs.outcome.result())
File "/Users/.../miniconda3/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/Users/.../miniconda3/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 61, in __call__
result = await fn(*args, **kwargs)
File "/Users/.../venv/lib/python3.10/site-packages/langchain_google_vertexai/llms.py", line 121, in _acompletion_with_retry_inner
return await llm.client.predict_async(prompt, **kwargs)
File "/Users/.../venv/lib/python3.10/site-packages/vertexai/language_models/_language_models.py", line 1399, in predict_async
prediction_response = await self._endpoint.predict_async(
File "/Users/.../venv/lib/python3.10/site-packages/google/cloud/aiplatform/models.py", line 1643, in predict_async
prediction_response = await self._prediction_async_client.predict(
File "/Users/.../venv/lib/python3.10/site-packages/google/cloud/aiplatform_v1/services/prediction_service/async_client.py", line 404, in predict
response = await rpc(
File "/Users/.../venv/lib/python3.10/site-packages/google/api_core/grpc_helpers_async.py", line 85, in __await__
response = yield from self._call.__await__()
File "/Users/.../venv/lib/python3.10/site-packages/grpc/aio/_call.py", line 299, in __await__
response = yield from self._call_response
File "/Users/.../miniconda3/lib/python3.10/asyncio/futures.py", line 285, in __await__
yield self # This tells Task to wait for completion.
RuntimeError: Task <Task pending name='Task-78' coro=<as_completed.<locals>.sema_coro() running at /Users/.../venv/lib/python3.10/site-packages/ragas/executor.py:38> cb=[as_completed.<locals>._on_completion() at /Users/.../miniconda3/lib/python3.10/asyncio/tasks.py:558]> got Future <Task pending name='Task-795' coro=<UnaryUnaryCall._invoke() running at /Users/.../venv/lib/python3.10/site-packages/grpc/aio/_call.py:568>> attached to a different loop
If you need any additional info please let me know! Thank you very much.
I forgot to add that the error was tested both in a jupyter notebook (using nest_asyncio.apply()) and a .py.
And the imports were both tested with
from langchain_google_vertexai import VertexAIEmbeddings, VertexAI
and
from langchain.chat_models import ChatVertexAI
from langchain.embeddings import VertexAIEmbeddings
Same problem for me with VertexAI stack in jupyter notebook. Slightly different models and embeddings used, but I'm getting this error right from the start of test set generation process.
@abetatos UPD
I've take a look into this PR, as a result of issue you've mentioned here.
I tried to clone that dev branch with fix, and install ragas from that source, but that hasn't worked. Then, I tried just to replace the src/ragas/executor.py file contents with the fixed code, presented in that PR, in my local (master) ragas package, and that worked - I'm able to generate test datasets with the VertexAI stack.
I still don't understand what is the root cause, and whether that fix is legit (since repo author hasn't really answered and merged that PR), but at least that works.