haystack
haystack copied to clipboard
create an example of how to improve a retriever on industry dataset
Goal: Showcase the Haystack evaluation metrics on an example that is close to what our users are trying to do, i.e. to improve a retriever in a RAG app.
The goal of this task is to find such an example, ideally finding:
- the RAG pipeline architecture
- the metrics to be used (if Haystack metrics are not finished yet, can use ragas/deepeval implementations for now). Here is the list of metrics that will be released in Haystack
- the experiment to show how the retriever improves and the improvement should show in the metrics: ideally after tweaking chunk size, or changing embedding model/size.
This does not need to be a finished tutorial or even code, just need some scrap code somewhere (colab?).
Depends on: https://github.com/deepset-ai/haystack/issues/7438