Timeout Error When Evaluating answer_similarity and answer_correctness in Ragas

[CZT0]

Describe the bug When attempting to evaluate answer_similarity and answer_correctness using the Ragas framework, I encounter a timeout error. While I can successfully retrieve metrics for context_relevancy and context_recall, the progress bar stalls for a long time and eventually times out when evaluating answer_similarity and answer_correctness.

Ragas version: ragas-0.0.23.dev37+g041b20c Python version: 3.11 DATA: 63521_1.json

Code to Reproduce

import os
import json
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import answer_similarity

from config import OPENAI_API_KEY, OPENAI_BASE_URL

os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY
os.environ["OPENAI_API_BASE"] = OPENAI_BASE_URL

def evaluate_ragas(file_path):
    with open(file_path, 'r', encoding='utf-8') as file:
        data = json.load(file)

    dataset = Dataset.from_dict(data)

    result = evaluate(
            dataset,
            metrics=[answer_similarity],
    )
    print(result)

if __name__ == '__main__':
    file_path = "63521_1.json"
    evaluate_ragas(file_path)

Error trace

/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import ChatOpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import AzureChatOpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import ChatVertexAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import AzureOpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import OpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/llms/__init__.py:548: LangChainDeprecationWarning: Importing LLMs from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.llms import VertexAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.embeddings import AzureOpenAIEmbeddings`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.embeddings import FastEmbedEmbeddings`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain/embeddings/__init__.py:29: LangChainDeprecationWarning: Importing embeddings from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.embeddings import OpenAIEmbeddings`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain_core/_api/deprecation.py:115: LangChainDeprecationWarning: The class `ChatOpenAI` was deprecated in LangChain 0.1.0 and will be removed in 0.2.0. Use langchain_openai.ChatOpenAI instead.
  warn_deprecated(
Evaluating:   0%|          | 0/5 [07:32<?, ?it/s]
Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 206, in connect_tcp
    sock = socket.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 851, in create_connection
    raise exceptions[0]
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 836, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
    yield
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 228, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request
    raise exc
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request
    response = connection.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectTimeout: [Errno 60] Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 877, in _request
    response = self._client.send(
               ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 901, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 929, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 966, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 1002, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 227, in handle_request
    with map_httpcore_exceptions():
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 206, in connect_tcp
    sock = socket.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 851, in create_connection
    raise exceptions[0]
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 836, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
    yield
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 228, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request
    raise exc
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request
    response = connection.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectTimeout: [Errno 60] Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 877, in _request
    response = self._client.send(
               ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 901, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 929, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 966, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 1002, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 227, in handle_request
    with map_httpcore_exceptions():
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 10, in map_exceptions
    yield
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 206, in connect_tcp
    sock = socket.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 851, in create_connection
    raise exceptions[0]
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/socket.py", line 836, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 60] Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 66, in map_httpcore_exceptions
    yield
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 228, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 268, in handle_request
    raise exc
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection_pool.py", line 251, in handle_request
    response = connection.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 99, in handle_request
    raise exc
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 76, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_backends/sync.py", line 205, in connect_tcp
    with map_exceptions(exc_map):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectTimeout: [Errno 60] Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 877, in _request
    response = self._client.send(
               ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 901, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 929, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 966, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_client.py", line 1002, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 227, in handle_request
    with map_httpcore_exceptions():
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/httpx/_transports/default.py", line 83, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout: [Errno 60] Operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/evaluation.py", line 176, in evaluate
    raise e
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/evaluation.py", line 159, in evaluate
    results = executor.results()
              ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/executor.py", line 118, in results
    raise e
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/executor.py", line 114, in results
    r = future.result()
        ^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/executor.py", line 36, in wrapped_callable
    return counter, callable(*args, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/metrics/base.py", line 75, in score
    raise e
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/metrics/base.py", line 71, in score
    score = self._score(row=row, callbacks=group_cm)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/metrics/_answer_similarity.py", line 70, in _score
    embeddings_1 = np.array(self.embeddings.embed_documents(ground_truths))
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain_community/embeddings/openai.py", line 668, in embed_documents
    return self._get_len_safe_embeddings(texts, engine=engine)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain_community/embeddings/openai.py", line 494, in _get_len_safe_embeddings
    response = embed_with_retry(
               ^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/langchain_community/embeddings/openai.py", line 116, in embed_with_retry
    return embeddings.client.create(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/resources/embeddings.py", line 106, in create
    return self._post(
           ^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 1088, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 853, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 884, in _request
    return self._retry_request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 958, in _retry_request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 884, in _request
    return self._retry_request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 958, in _retry_request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/openai/_base_client.py", line 893, in _request
    raise APITimeoutError(request=request) from err
openai.APITimeoutError: Request timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jellow/code/python/summary_tree/data_process/SQuALITY/计算ragas.py", line 40, in <module>
    evaluate_ragas("63521", result_id="1")
  File "/Users/jellow/code/python/summary_tree/data_process/SQuALITY/计算ragas.py", line 24, in evaluate_ragas
    result = evaluate(
             ^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/evaluation.py", line 178, in evaluate
    result = Result(
             ^^^^^^^
  File "<string>", line 6, in __init__
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/ragas/evaluation.py", line 207, in __post_init__
    for cn in self.scores[0].keys():
              ~~~~~~~~~~~^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 2795, in __getitem__
    return self._getitem(key)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 2779, in _getitem
    pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/datasets/formatting/formatting.py", line 583, in query_table
    _check_valid_index_key(key, size)
  File "/opt/homebrew/anaconda3/envs/summary_tree/lib/python3.11/site-packages/datasets/formatting/formatting.py", line 526, in _check_valid_index_key
    raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
IndexError: Invalid key: 0 is out of bounds for size 0

Expected behavior I expected the answer_similarity and answer_correctness evaluations to succeed without a timeout error, similar to the other metrics.

Additional context In my use of Ragas for evaluation, other metrics like context_relevancy and context_recall work normally. The issue arises solely when evaluating answer_similarity and answer_correctness.