[Bug] context_recall metric failing when using custom dataset

[deshraj]

Describe the bug Hi, I am trying to setup evaluation on my custom dataset using Ragas but getting the following error when trying to evaluate for context_recall metric:

evaluating with [context_recall]
 17%|████████▊                                            | 1/6 [00:24<02:02, 24.42s/it]
Traceback (most recent call last):

  File ~/Projects/embedchain/.venv/lib/python3.11/site-packages/IPython/core/interactiveshell.py:3550 in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  Cell In[50], line 3
    result2 = evaluate(

  File ~/Projects/embedchain/.venv/lib/python3.11/site-packages/ragas/evaluation.py:105 in evaluate
    scores.append(metric.score(dataset).select_columns(metric.name))

  File ~/Projects/embedchain/.venv/lib/python3.11/site-packages/ragas/metrics/base.py:76 in score
    score = self._score_batch(dataset.select(batch), callbacks=group)

  File ~/Projects/embedchain/.venv/lib/python3.11/site-packages/ragas/metrics/_context_recall.py:120 in _score_batch

  File <string>:13
    "statement_3": "Load, split, and embed your documents: loader = TextLoader('state_of_the_union.txt'), documents = loader.load(), text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0, separator='
                   ^
SyntaxError: unterminated string literal (detected at line 13)

Ragas version: 0.0.21 Python version: 3.11

Code to Reproduce Use the following code to reproduce. Unfortunately I cannot share the dataset here.

from ragas import evaluate

result2 = evaluate(
    dataset,
    metrics=[
        context_recall,
    ],
)

Error trace Shared above.

Expected behavior Expect to see he context_recall score.

[shahules786]

Thanks for sharing @deshraj , Can you retry after installing ragas from source pip install git+https://github.com/explodinggradients/ragas.git

[arm-diaz]

Hi, I am getting NaN for almost all metrics using Bedrock.

[arm-diaz]

User's warning with invalid json

 0%|          | 0/2 [00:00<?, ?it/s]/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/utils.py:29: UserWarning: Invalid json: Expecting value: line 1 column 2 (char 1)
  warnings.warn(f"Invalid json: {e}")
/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/utils.py:29: UserWarning: Invalid json: Expecting value: line 1 column 2 (char 1)
  warnings.warn(f"Invalid json: {e}")
 50%|█████     | 1/2 [00:19<00:19, 19.78s/it]/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/utils.py:29: UserWarning: Invalid json: Expecting value: line 1 column 2 (char 1)
  warnings.warn(f"Invalid json: {e}")
/home/ec2-user/anaconda3/envs/python3/lib/python3.10/site-packages/ragas/utils.py:29: UserWarning: Invalid json: Expecting value: line 1 column 2 (char 1)
  warnings.warn(f"Invalid json: {e}")

[RuiNov1st]

Have you fixed it already ? I have the same problem.

[jjmachan]

@arm-diaz we need to custumize the prompt for bedrock actually. there is #298 which has the correct prompts but we need to fix that

@RuiNov1st are you facing the issue with bedrock?

@deshraj has this issue been resolved for you when installing from source?