ragas
ragas copied to clipboard
Ensure question variedness when using ResponseRelevancy
- [x] I have checked the documentation and related resources and couldn't resolve my bug.
Describe the bug ResponseRelevancy does not guarantee varied questions. This makes strictness effectively pointless.
Ragas version: 0.2.14 Python version: 3.12.3
Code to Reproduce Run ragas.evaluate using ragas.metrics.answer_relevancy as metric.
Expected behavior Generating varied questions.
Proposed solutions
- Generate all questions at the same time, relying on the LLM to respect the request for variedness.
- Keep generating questions, with a limited amount of retries, until a unique set of questions is generated. This has the potential to get stuck in a loop and should only be implemented if the first solution is not possible.