Alternative for OpenAI in evaluation

[onoph]

Hi,

I'm new to haystack and I think it's a very pleasant tool. I've been searching for a while now (mainly here: https://docs.haystack.deepset.ai/v2.0/reference/integrations-ragas) but I didn't find an answer to this question : it is possible to use Ragas with other LLMs ? Like Ollama (thinking to mixtral), or any other provider ?

We would like to keep the data private

[davidsbatista]

Hi @onoph

It's not possible at the moment, but it's something on our radar and what we want to enable in future versions.

[onoph]

@davidsbatista Thanks for your answer and good luck for the project. I'm starting to like a lot Haystack 😄

[satyaloka93]

I would also like alternative, we shouldn't constrain to a dependency of paid api services, and particularly sending our data to these services. Any decent local model should be able to self evaluate.

[mrm1001]

@julian-risch : something to consider when doing the LLM-based metrics: https://github.com/deepset-ai/haystack/issues/7022

[DevYourDreams]

@mrm1001 thanks I'm going to explore this. I've seen in the deepEval official doc that we can use any LLM to evaluate the content

Edit for people who will be interested by this topic : Currently I don't see any evaluation for a custom LLM. It seems only GTP models (and a few other online models) are supported regarding this page : https://docs.haystack.deepset.ai/docs/model-based-evaluation#evaluation-framework-integrations

I consider using DeepEval natively : https://docs.confident-ai.com/docs/metrics-introduction

[gustavo-has-stone]

Is it currently possible to choose which OpenAI model should be used by RAGAS?

[sadaisystems]

Is it possible to choose OpenAI model for RAGAS? It seems like the code itself doesn't support this. Am i wrong?

[julian-risch]

There is a new version of the Ragas-Haystack integration that makes use of a HaystackLLMWrapper that allows using different LLM providers: https://haystack.deepset.ai/integrations/ragas