haystack icon indicating copy to clipboard operation
haystack copied to clipboard

Showcase Haystack evaluations on an industry dataset

Open mrm1001 opened this issue 11 months ago • 0 comments

User story

I would like to learn how to apply Haystack core evaluations to improve my RAG pipeline, with an example on how to improve my retriever component, on an industry dataset.

Sub-tasks:

  • create an example of how to improve Haystack evaluation metrics by tweaking: chunk size and/or embedding model (e.g. context size). Using: semantic similarity on answers and LLM-based metric context relevance.

More context here: https://www.notion.so/deepsetai/Evaluation-1521712b928d4142828232f2df136856?pvs=4

### Tasks
- [ ] https://github.com/deepset-ai/haystack/issues/7438
- [ ] https://github.com/deepset-ai/haystack/issues/6790

mrm1001 avatar Mar 22 '24 09:03 mrm1001