haystack icon indicating copy to clipboard operation
haystack copied to clipboard

feat: Add `embed_meta_fields` to the Ranker nodes

Open sjrl opened this issue 2 years ago • 0 comments

Is your feature request related to a problem? Please describe. I find the embed_meta_fields option of the EmbeddingRetriever to be extremely helpful in retrieving relevant results for large files that have been split up into many smaller Haystack Documents.

Recently, we have been using Cross-Encoder-based Rankers in Haystack to further improve the retrieval results (e.g. by reranking documents from multiple sources such as BM25 and Embedding Retrieval). However, the Rankers in Haystack do not have an option like embed_meta_fields so the final results can end up being less relevant since the model is missing information that the Embedding Retriever received.

Describe the solution you'd like I think it would be great to add embed_meta_fields to the Rankers in Haystack as well. This would help boost relevant results based on meta data.

Describe alternatives you've considered There isn't really an alternative to this other than not using the Ranker node.

sjrl avatar Jul 13 '23 13:07 sjrl