ragas icon indicating copy to clipboard operation
ragas copied to clipboard

Ensure question variedness when using ResponseRelevancy

Open aabanic opened this issue 9 months ago • 0 comments

  • [x] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug ResponseRelevancy does not guarantee varied questions. This makes strictness effectively pointless.

Ragas version: 0.2.14 Python version: 3.12.3

Code to Reproduce Run ragas.evaluate using ragas.metrics.answer_relevancy as metric.

Expected behavior Generating varied questions.

Proposed solutions

  • Generate all questions at the same time, relying on the LLM to respect the request for variedness.
  • Keep generating questions, with a limited amount of retries, until a unique set of questions is generated. This has the potential to get stuck in a loop and should only be implemented if the first solution is not possible.

aabanic avatar Mar 24 '25 21:03 aabanic