explodinggradients
Nvidia Accuracy calculation code is not consistent with the docs
[ x ] I have checked the documentation and related resources and couldn't resolve my bug.
Describe the bug In the Nvidia Accuracy metric calculation, the User Answer and Reference Answer were supposed to be swapped (according to docs) in steps 1 and 2. However, it's not really swapping, as both titles and values are being swapped.
Ragas version: 0.2.14
Hi @vitorsabbagh, are you just referring to the example in the docs?
the example calculation is not the best example because both the response and reference is the same.
If we look under the hood, it swapping is happening.
- https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_nv_metrics.py#L109
formatted_prompt = StringPromptValue(
text=self.template_accuracy1.format(
query=sample.user_input,
answer0="User Answer",
answer1="Reference Answer",
sentence_inference=sample.response,
sentence_true=sample.reference,
)
)
- https://github.com/explodinggradients/ragas/blob/main/src/ragas/metrics/_nv_metrics.py#L132
formatted_prompt = StringPromptValue(
text=self.template_accuracy2.format(
query=sample.user_input,
answer0="Reference Answer",
answer1="User Answer",
sentence_inference=sample.reference,
sentence_true=sample.response,
)
)
notice answer0 and answer1 they are getting swapped.
Hi @sahusiddharth, you're right! However, notice that sentence_inference and sentence_true are also being swapped. It's a double swap.
Yes @vitorsabbagh, you are correct thank-you letting us know. I will raise a PR to fix this.
I believe the implementation is right. The docs rather needs update to clarify the double swap.
@shahules786 Could you please verify once?
