Message 'o statements were generated from the answer' was sent when evaluating and the result is like this: Faithfulness:nan

[francescofan]

[ ] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug Message 'o statements were generated from the answer' was sent when evaluating and the result is like this: Faithfulness:nan

Ragas version:0.1.20 Python version:3.11.9 System: Windows 10 IDE: Visual source code

Code to Reproduce

# data contais a pack of chinese question/answer/contexts/ground_truth values
nest_asyncio.apply()
dataset = Dataset.from_dict(data)
score = evaluate(dataset,metrics=[faithfulness,context_recall,context_precision,answer_correctness,answer_similarity,answer_relevancy],llm=deepseek_llm,embeddings=hp_embeddings)

Error trace Evaluating: 17%|█████████████████████ | 2/12 [00:01<00:07, 1.36it/s]No statements were generated from the answer. Evaluating: 58%|█████████████████████████████████████████████████████████████████████████▌ | 7/12 [00:04<00:02, 2.07it/s]No statements were generated from the answer. Evaluating: 92%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 11/12 [00:06<00:00, 2.41it/s]Exception raised in Job[1]: TimeoutError() Evaluating: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [03:00<00:00, 15.00s/it]

Expected behavior No error or warning or abnormal message and get the correct result. No nan metirc.

Additional context Add any other context about the problem here.

[dosubot[bot]]

The message "No statements were generated from the answer" and the "Faithfulness:nan" result occur when no statements are parsed from the answer. This can happen if the sentence segmentation process doesn't properly break down the response into simpler statements. Ensure that the answer is correctly segmented into statements without pronouns ^[1].

A similar issue was discussed in this thread, where it was suggested to check the sentence segmentation process to ensure it is correctly breaking down the response ^[2].

Additionally, the TimeoutError might be due to the timeout parameter in the evaluation process. If a metric evaluation takes longer than the specified timeout, this error can occur ^[3].

To continue talking to Dosu, mention @dosu.

[qpwo2468]

I have also encountered the same problem, who can help answer it? Please

[francescofan]

@qpwo2468 do you have any solutions?

[mariauhl]

@qpwo2468 @francescofan did you solve it?

[jjmachan]

hey @mariauhl did you check out the latest version? We recently merged something that should fix this and it will be released this week but could you check out the main branch and see if it works for you?

you can install it from source as mentioned here: https://docs.ragas.io/en/stable/getstarted/install/