answer_relevancy score decreased drasctically

[jinnoh47]

[ ] I checked the documentation and related resources and couldn't find an answer to my question.

Your Question The code is identical only difference is the packages version ex) ragas, langchain it answer_relevancy used to be above 0.8 now it is somewhere around 0.7 I have no idea what is the issue.

[dosubot[bot]]

I found a similar open issue titled Bug in Answer Similarity Calculation, which might be related to your problem. This issue is currently closed and was last updated on June 01, 2024 ^[1].

Regarding the decrease in the answer_relevancy score, it's important to consider the dependencies and their versions, as they can influence the score. However, I couldn't find specific information on changes in recent versions of the ragas package that could affect the answer_relevancy score ^[2].

One parameter that could impact the score is the strictness parameter in the ResponseRelevancy class. It determines the number of questions generated per answer, which can affect the score calculation ^[2].

If you haven't already, you might want to check the versions of the ragas and langchain packages you're using, as changes in these packages could potentially affect the score.

To continue talking to Dosu, mention @dosu.

[jinnoh47]

what was the version before parameter change and after? I'm using ragas 0.1.13 version

[dosubot[bot]]

I wasn't able to find specific information on the version changes related to the strictness parameter or any other parameter changes in the ragas package that could affect the answer_relevancy score. You might want to check the release notes or changelogs of the ragas package for detailed information on version changes.

To continue talking to Dosu, mention @dosu.