explodinggradients
New metrics about answers
In our practical use, we have found that in addition to the existing metrics of RAGAS, we also place great importance on metrics related to the quality of the answers generated by LLM, such as the Semantic Diversity and Lexical Diversity of the answers. These metrics that measure whether the answer is comprehensive and whether it elaborately explains the questions posed by the users are also an important reference for our evaluation. Moreover, we have already implemented this capability based on the existing framework of RAGAS. With Semantic Diversity, We referred to the paper https://arxiv.org/pdf/2205.01497.pdf, and https://arxiv.org/pdf/2307.04626.pdf with Lexical Diversity. Could you please inform me if the community is considering adding metrics to RAGAS? If needed, we are very willing to share our code.
Hi,
It will be great help to the community if you can share the code.
Regards, Saby
Hey @prime-7 Yes of course, we are always open to adding new metrics that help the community. Please do a PR on it, if you want to chat with me before that feel free to book a slot here.
Just briefly checkout out both papers, would love to chat and know your use-case for the metrics. @prime-7
OK, please give me some time, there are some works to be done, such as matching latest version, changing the language of prompt in English.