explodinggradients
Exception in thread Thread-4: `asyncio.exceptions.CancelledError` & `asyncio.exceptions.TimeoutError`
I have checked the documentation and related resources and couldn't resolve my bug.
Describe the bug I am trying to run the template code from the Github ReadMe page. My code is below:
from datasets import Dataset
import os
from ragas import evaluate
from ragas.metrics import faithfulness
os.environ["OPENAI_API_KEY"] = "sk-proj-*****B3" <HIDDEN>
data_samples = {
'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
'contexts' : [['The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles,'],
['The Green Bay Packers...Green Bay, Wisconsin.','The Packers compete...Football Conference']],
'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
}
dataset = Dataset.from_dict(data_samples)
score = evaluate(dataset,metrics=[faithfulness])
score.to_pandas()
Ragas version: I use the latest one. Python version: 3.10.14.
Error trace The error I encountered is below: (with my username hidden in the file paths)
**Exception in thread Thread-4:**
Traceback (most recent call last):
File "/Users/*****/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/metrics/_faithfulness.py", line 248, in _ascore
statements = await self.llm.generate(
File "/Users/*****/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/llms/base.py", line 93, in generate
return await agenerate_text_with_retry(
File "/Users/*****/anaconda3/envs/ragas/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 189, in async_wrapped
return await copy(fn, *args, **kwargs)
File "/Users/*****/anaconda3/envs/ragas/lib/python3.10/site-packages/tenacity/asyncio/__init__.py", line 121, in __call__
await self.sleep(do) # type: ignore[misc]
File "/Users/*****/anaconda3/envs/ragas/lib/python3.10/asyncio/tasks.py", line 605, in sleep
return await future
**asyncio.exceptions.CancelledError**
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/*****/anaconda3/envs/ragas/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
return fut.result()
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/*****/anaconda3/envs/ragas/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
...
score = await asyncio.wait_for(
File "/Users/*****/anaconda3/envs/ragas/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
raise exceptions.TimeoutError() from exc
**asyncio.exceptions.TimeoutError**
Additional context And the error report of the code is here:
---------------------------------------------------------------------------
ExceptionInRunner Traceback (most recent call last)
Cell In[1], [line 18](vscode-notebook-cell:?execution_count=1&line=18)
[8](vscode-notebook-cell:?execution_count=1&line=8) data_samples = {
[9](vscode-notebook-cell:?execution_count=1&line=9) 'question': ['When was the first super bowl?', 'Who won the most super bowls?'],
[10](vscode-notebook-cell:?execution_count=1&line=10) 'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'],
(...)
[13](vscode-notebook-cell:?execution_count=1&line=13) 'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times']
[14](vscode-notebook-cell:?execution_count=1&line=14) }
[16](vscode-notebook-cell:?execution_count=1&line=16) dataset = Dataset.from_dict(data_samples)
---> [18](vscode-notebook-cell:?execution_count=1&line=18) score = evaluate(dataset,metrics=[faithfulness])
[19](vscode-notebook-cell:?execution_count=1&line=19) score.to_pandas()
File ~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:255, in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, is_async, run_config, raise_exceptions, column_map)
[252](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:252) if not evaluation_group_cm.ended:
[253](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:253) evaluation_rm.on_chain_error(e)
--> [255](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:255) raise e
[256](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:256) else:
[257](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:257) result = Result(
[258](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:258) scores=Dataset.from_list(scores),
[259](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:259) dataset=dataset,
[260](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:260) binary_columns=binary_metrics,
[261](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:261) )
File ~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:237, in evaluate(dataset, metrics, llm, embeddings, callbacks, in_ci, is_async, run_config, raise_exceptions, column_map)
...
--> [237](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:237) raise ExceptionInRunner()
[239](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:239) # convert results to dataset_like
[240](https://file+.vscode-resource.vscode-cdn.net/Users/*****/Library/CloudStorage/OneDrive-DukeUniversity/029.%20SRS%202024/Paul%20Weng%20_%20LLM/Evaluation/~/anaconda3/envs/ragas/lib/python3.10/site-packages/ragas/evaluation.py:240) for i, _ in enumerate(dataset):
ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
Can anyone help me out? Thanks!
Hey, @larry-ziyue-yin! I'm here to help you with your bug. I can assist with solving issues, answering questions, and becoming a contributor. Let's get this sorted out!
To resolve the asyncio.exceptions.CancelledError and asyncio.exceptions.TimeoutError when running the template code from the Ragas library's GitHub ReadMe page, you can handle these exceptions within the _aresults method of the Runner class. Here is the modified code with the necessary exception handling:
async def _aresults(self) -> t.List[t.Any]:
results = []
for future in tqdm(
self.futures,
desc=self.desc,
total=len(self.jobs),
# whether you want to keep the progress bar after completion
leave=self.keep_progress_bar,
):
r = (-1, np.nan)
try:
r = await future
except MaxRetriesExceeded as e:
logger.warning(f"max retries exceeded for {e.evolution}")
except asyncio.exceptions.CancelledError as e:
logger.warning("Task was cancelled")
if self.raise_exceptions:
raise e
except asyncio.exceptions.TimeoutError as e:
logger.warning("Task timed out")
if self.raise_exceptions:
raise e
except Exception as e:
if self.raise_exceptions:
raise e
else:
logger.error(
"Runner in Executor raised an exception", exc_info=True
)
results.append(r)
return results
This code adds specific handling for asyncio.exceptions.CancelledError and asyncio.exceptions.TimeoutError, logging a warning message for each and re-raising the exception if self.raise_exceptions is set to True [1].
Additionally, ensure that there are no null values in the ground_truth column of your dataset, as this can also cause issues. This was a known issue in a similar context [2].
To continue talking to Dosu, mention @dosu.
![dosubot[bot] avatar](https://avatars.githubusercontent.com/in/324583?v=4 "Published by a pub.dev verified publisher"){kind=small}
I also encounter this problem when evaluating, but I don't want to change ragas codebase inside. I found increasing the time-related parameters(timeout, thread_timeout, max_wait, max_retries) in run_config will have a certain effect, but I don't know if it can solve the problem fundamentally.
result = evaluate(
dataset,
llm=model,
embeddings=embeddings,
metrics=[
context_precision,
faithfulness,
answer_relevancy,
context_recall,
],
run_config=RunConfig(thread_timeout=thread_timeout)
)
Is there a reasonable explanation for this problem? Guess: I adapt evaluation metrics to Chinese, does the segmenting and processing time of Chinese increase the processing time compared to English?
context_precision,
Hello, I have the same problem, do you have a new solution?
@jmgu0212 evaluate on current master does not accept run_config. See my PR https://github.com/explodinggradients/ragas/pull/1085
I also encounter this problem when evaluating, but I don't want to change ragas codebase inside. I found increasing the time-related parameters(timeout, thread_timeout, max_wait, max_retries) in run_config will have a certain effect, but I don't know if it can solve the problem fundamentally.
result = evaluate( dataset, llm=model, embeddings=embeddings, metrics=[ context_precision, faithfulness, answer_relevancy, context_recall, ], run_config=RunConfig(thread_timeout=thread_timeout) )Is there a reasonable explanation for this problem? Guess: I adapt evaluation metrics to Chinese, does the segmenting and processing time of Chinese increase the processing time compared to English?
Hello, I would like to ask about the thread_timeout setting in your run_config=RunConfig(thread_timeout=thread_timeout). My configuration is as follows: run_config = ragas.RunConfig(timeout=60, max_retries=3, max_wait=20). However, when calculating metrics, such as context_relevancy, the final results are all NA. I wonder why this is happening and if there are any solutions. Is it because the timeout I set is too short?
I also encounter this problem when evaluating, but I don't want to change ragas codebase inside. I found increasing the time-related parameters(timeout, thread_timeout, max_wait, max_retries) in run_config will have a certain effect, but I don't know if it can solve the problem fundamentally.
result = evaluate( dataset, llm=model, embeddings=embeddings, metrics=[ context_precision, faithfulness, answer_relevancy, context_recall, ], run_config=RunConfig(thread_timeout=thread_timeout) )Is there a reasonable explanation for this problem? Guess: I adapt evaluation metrics to Chinese, does the segmenting and processing time of Chinese increase the processing time compared to English?
Hello, I would like to ask about the thread_timeout setting in your run_config=RunConfig(thread_timeout=thread_timeout). My configuration is as follows: run_config = ragas.RunConfig(timeout=60, max_retries=3, max_wait=20). However, when calculating metrics, such as context_relevancy, the final results are all NA. I wonder why this is happening and if there are any solutions. Is it because the timeout I set is too short?
I wonder if you also adapt evaluation metrics to Chinese? If yes, you need to deal with the language option fixed in source code. Take context_relevancy as an example, you will find seg = pysbd.Segmenter(language="en", clean=False).
I also encounter this problem when evaluating, but I don't want to change ragas codebase inside. I found increasing the time-related parameters(timeout, thread_timeout, max_wait, max_retries) in run_config will have a certain effect, but I don't know if it can solve the problem fundamentally.
result = evaluate( dataset, llm=model, embeddings=embeddings, metrics=[ context_precision, faithfulness, answer_relevancy, context_recall, ], run_config=RunConfig(thread_timeout=thread_timeout) )Is there a reasonable explanation for this problem? Guess: I adapt evaluation metrics to Chinese, does the segmenting and processing time of Chinese increase the processing time compared to English?
Hello, I would like to ask about the thread_timeout setting in your run_config=RunConfig(thread_timeout=thread_timeout). My configuration is as follows: run_config = ragas.RunConfig(timeout=60, max_retries=3, max_wait=20). However, when calculating metrics, such as context_relevancy, the final results are all NA. I wonder why this is happening and if there are any solutions. Is it because the timeout I set is too short?
I wonder if you also adapt evaluation metrics to Chinese? If yes, you need to deal with the language option fixed in source code. Take context_relevancy as an example, you will find
seg = pysbd.Segmenter(language="en", clean=False).
Thank you very much for your answer. The data set I use is given in Chinese form. So far, I only run one index context_relevancy. I found context_relevancy source code set to seg = pysbd.Segmenter(language="zh", clean=False), but it still ran with the following error: Traceback (most recent call last):
File "C:\Program Files\Python311\Lib\site-packages\httpx_transports\default.py", line 69, in map_httpcore_exceptions
yield
File "C:\Program Files\Python311\Lib\site-packages\httpx_transports\default.py", line 373, in handle_async_request
resp = await self._pool.handle_async_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\httpcore_async\connection_pool.py", line 216, in handle_async_request
raise exc from None
File "C:\Program Files\Python311\Lib\site-packages\httpcore_async\connection_pool.py", line 196, in handle_async_request
response = await connection.handle_async_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\httpcore_async\http_proxy.py", line 317, in handle_async_request
stream = await stream.start_tls(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\httpcore_async\http11.py", line 383, in start_tls
return await self._stream.start_tls(ssl_context, server_hostname, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\httpcore_backends\anyio.py", line 68, in start_tls
with map_exceptions(exc_map):
File "C:\Program Files\Python311\Lib\contextlib.py", line 158, in exit
self.gen.throw(typ, value, traceback)
File "C:\Program Files\Python311\Lib\site-packages\httpcore_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Program Files\Python311\Lib\site-packages\openai_base_client.py", line 1548, in _request
response = await self._client.send(
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\httpx_client.py", line 1661, in send
response = await self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\httpx_client.py", line 1689, in _send_handling_auth
response = await self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\httpx_client.py", line 1726, in _send_handling_redirects
response = await self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\httpx_client.py", line 1763, in _send_single_request
response = await transport.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\httpx_transports\default.py", line 372, in handle_async_request
with map_httpcore_exceptions():
File "C:\Program Files\Python311\Lib\contextlib.py", line 158, in exit
self.gen.throw(typ, value, traceback)
File "C:\Program Files\Python311\Lib\site-packages\httpx_transports\default.py", line 86, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Program Files\Python311\Lib\site-packages\ragas\executor.py", line 104, in wrapped_callable_async
result = await callable(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\ragas\metrics\base.py", line 134, in ascore
raise e
File "C:\Program Files\Python311\Lib\site-packages\ragas\metrics\base.py", line 127, in ascore
score = await asyncio.wait_for(
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\asyncio\tasks.py", line 489, in wait_for
return fut.result()
^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\ragas\metrics_context_relevancy.py", line 77, in ascore
result = await self.llm.generate(
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\ragas\llms\base.py", line 93, in generate
return await agenerate_text_with_retry(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\tenacity\asyncio_init.py", line 189, in async_wrapped
return await copy(fn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\tenacity\asyncio_init_.py", line 111, in call
do = await self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\tenacity\asyncio_init_.py", line 153, in iter
result = await action(retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\tenacity_utils.py", line 99, in inner
return call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\tenacity_init_.py", line 398, in
response = await self.async_client.create(**payload)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\openai\resources\chat\completions.py", line 1289, in create
return await self._post(
^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\openai_base_client.py", line 1816, in post
return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\openai_base_client.py", line 1514, in request
return await self._request(
^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\openai_base_client.py", line 1572, in _request
return await self._retry_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\openai_base_client.py", line 1641, in _retry_request
return await self._request(
^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\openai_base_client.py", line 1572, in _request
return await self._retry_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\openai_base_client.py", line 1641, in _retry_request
return await self._request(
^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\openai_base_client.py", line 1582, in _request
raise APIConnectionError(request=request) from err
openai.APIConnectionError: Connection error. Do you have a better solution? Thank you very much
@djanito @kmr666 @JANHMS @jmgu0212 @larry-ziyue-yin my sincere apologies for the delay in responding but this will be addressed with #1156 do check that out
I also encounter this problem when evaluating, but I don't want to change ragas codebase inside. I found increasing the time-related parameters(timeout, thread_timeout, max_wait, max_retries) in run_config will have a certain effect, but I don't know if it can solve the problem fundamentally.
result = evaluate( dataset, llm=model, embeddings=embeddings, metrics=[ context_precision, faithfulness, answer_relevancy, context_recall, ], run_config=RunConfig(thread_timeout=thread_timeout) )Is there a reasonable explanation for this problem? Guess: I adapt evaluation metrics to Chinese, does the segmenting and processing time of Chinese increase the processing time compared to English?
Hello, I would like to ask about the thread_timeout setting in your run_config=RunConfig(thread_timeout=thread_timeout). My configuration is as follows: run_config = ragas.RunConfig(timeout=60, max_retries=3, max_wait=20). However, when calculating metrics, such as context_relevancy, the final results are all NA. I wonder why this is happening and if there are any solutions. Is it because the timeout I set is too short?
yes,your timeout is too short , just make it as long as possible ,such as tiemout=1000,this can fix your nan problem
