ragas icon indicating copy to clipboard operation
ragas copied to clipboard

is_async missing in context_relevancy in ragas 0.1.8

Open abetatos opened this issue 1 year ago • 0 comments

In ragas 0.1.8 in _context_relevancy.py

    async def _ascore(self, row: t.Dict, callbacks: Callbacks, is_async: bool) -> float:
        assert self.llm is not None, "LLM is not initialized"

        if self.show_deprecation_warning:
            logger.warning(
                "The 'context_relevancy' metric is going to be deprecated soon! Please use the 'context_precision' metric instead. It is a drop-in replacement just a simple search and replace should work."  # noqa
            )

        question, contexts = row["question"], row["contexts"]
        result = await self.llm.generate(
            self.context_relevancy_prompt.format(
                question=question, context="\n".join(contexts)
            ),
            callbacks=callbacks
        )
        return self._compute_score(result.generations[0][0].text, row)

I think self.llm.generate is missing is_async=is_async. I don't know if this is a bug or not, but found it weird it was the only one who lacked the argument and in my setup is failing only this metric.

Thank you in advance!

abetatos avatar May 23 '24 11:05 abetatos