ragas
ragas copied to clipboard
Seems like `context_recall` is not working in Azure OpenAI
Describe the bug
Seems like context_recall is not working in Azure OpenAI
Ragas version: v0.1.1 Python version:
Code to Reproduce this was from a user, but will try to reproduce this with amnesty_qa
Error trace
WARNING:ragas.metrics._context_recall:Invalid JSON response. Expected dictionary with key 'Attributed'
WARNING:ragas.metrics._context_recall:Invalid JSON response. Expected dictionary with key 'Attributed'
/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py:276: RuntimeWarning: Mean of empty slice
value = np.nanmean(self.scores[cn])
{'context_precision': 0.5000, 'faithfulness': 0.5417, 'answer_relevancy': 0.9653, 'context_recall': nan, 'harmfulness': 0.0000, 'context_relevancy': 1.000
I'm facing this issue as well