ragas icon indicating copy to clipboard operation
ragas copied to clipboard

Seems like `context_recall` is not working in Azure OpenAI

Open jjmachan opened this issue 1 year ago • 1 comments

Describe the bug Seems like context_recall is not working in Azure OpenAI

Ragas version: v0.1.1 Python version:

Code to Reproduce this was from a user, but will try to reproduce this with amnesty_qa

Error trace

WARNING:ragas.metrics._context_recall:Invalid JSON response. Expected dictionary with key 'Attributed'
WARNING:ragas.metrics._context_recall:Invalid JSON response. Expected dictionary with key 'Attributed'
/usr/local/lib/python3.10/dist-packages/ragas/evaluation.py:276: RuntimeWarning: Mean of empty slice
  value = np.nanmean(self.scores[cn])

{'context_precision': 0.5000, 'faithfulness': 0.5417, 'answer_relevancy': 0.9653, 'context_recall': nan, 'harmfulness': 0.0000, 'context_relevancy': 1.000

jjmachan avatar Feb 17 '24 04:02 jjmachan

I'm facing this issue as well

cb-bharathvaj avatar Mar 13 '24 04:03 cb-bharathvaj