haystack
haystack copied to clipboard
Showcase Haystack evaluations on an industry dataset
User story
I would like to learn how to apply Haystack core evaluations to improve my RAG pipeline, with an example on how to improve my retriever component, on an industry dataset.
Sub-tasks:
- create an example of how to improve Haystack evaluation metrics by tweaking: chunk size and/or embedding model (e.g. context size). Using: semantic similarity on answers and LLM-based metric context relevance.
More context here: https://www.notion.so/deepsetai/Evaluation-1521712b928d4142828232f2df136856?pvs=4
### Tasks
- [ ] https://github.com/deepset-ai/haystack/issues/7438
- [ ] https://github.com/deepset-ai/haystack/issues/6790