ragas Context_recall returns only nan when using gpt-4-0125-preview and gpt-4-1106-preview

Describe the bug When using gpt-4-0125-preview and gpt-4-1106-preview as llm I obtain only nan values for the context_recall metrics.

Ragas version: 0.1.5 Python version: 3.11.7

Code to Reproduce

result_ragas = evaluate(
    data,
    metrics=[
        context_recall
    ],
    llm=ChatOpenAI(model_name="gpt-4-0125-preview", temperature=0.0)
)

Mar 18 '24 14:03 davideftimov

same issue for me!

Mar 18 '24 14:03 weissenbacherpwc

The problem comes from the parsing of the output. gpt-4-0125-preview always responds with: {"classification": []}

Changing this line in _context_recall.py response = response if isinstance(response, list) else [response] with response = response['classification'] solves the problem

Mar 21 '24 08:03 davideftimov

The problem comes from the parsing of the output. gpt-4-0125-preview always responds with: {"classification": []}

Changing this line in _context_recall.py response = response if isinstance(response, list) else [response] with response = response['classification'] solves the problem

no, it can not be solved, because the dict generated by GPT always not strictly follow the example, and randomly generated, so the context recall and context precision should not be generated in this manner

Mar 27 '24 11:03 frischzenger

Hey @frischzenger We are about to merge a release that would fix most of the NaN related issues. #807

Mar 27 '24 17:03 shahules786