haystack icon indicating copy to clipboard operation
haystack copied to clipboard

create an example of how to improve a retriever on industry dataset

Open mrm1001 opened this issue 10 months ago • 0 comments

Goal: Showcase the Haystack evaluation metrics on an example that is close to what our users are trying to do, i.e. to improve a retriever in a RAG app.

The goal of this task is to find such an example, ideally finding:

  • the RAG pipeline architecture
  • the metrics to be used (if Haystack metrics are not finished yet, can use ragas/deepeval implementations for now). Here is the list of metrics that will be released in Haystack
  • the experiment to show how the retriever improves and the improvement should show in the metrics: ideally after tweaking chunk size, or changing embedding model/size.

This does not need to be a finished tutorial or even code, just need some scrap code somewhere (colab?).

Depends on: https://github.com/deepset-ai/haystack/issues/7438

mrm1001 avatar Mar 28 '24 13:03 mrm1001