Add a ranker component that uses an LLM to rerank documents

[sjrl]

Describe the solution you'd like I’d like to add a new ranker component that leverages a LLM to rerank retrieved documents based on their relevance to the query. This would better assess the quality of the top-ranked documents, helping ensure that only relevant results are given to the LLM to answer the question.

Additionally, having an ability for the LLM to choose how many documents to keep would also be nice. A sort of dynamic top-k if you will.

Additional context We have started to employ this for some clients especially in situations where we need to provide extensive references. Basically for a given answer we need to provide all relevant documents that support the answer text. Having one reference in these situations is not enough. As a result in these situations we are willing to pay the extra cost to use an LLM to rerank and only keep the most relevant documents.

[srini047]

@sjrl I would like to take up this issue. I find rank_llm as a powerful toolkit that addresses the usecase. Shall I proceed with this? What are your thoughts?

[sjrl]

Hey @srini047 thanks for your interest! rank_llm certainly looks like an interesting tool. However, I think a good first version of this for Haystack would be to utilize our existing ChatGenerators to power the LLM calling. Then this new component would wrap the ChatGnerator to handle the input and output requirements (ie docs as input and docs as output).

[srini047]

Hi @sjrl ,

Currently I have a implementation here (of how reranker using haystack generator would look): https://colab.research.google.com/drive/1t9ohLid1DEk6E49LsQaN9jqoDzLexmU-?usp=sharing

To set the understanding right, I need to have these parameters to the LLMRanker component:

• User query mandatory
• Default generator (say OpenAIGenerator i.e. Optional)
• prompt (with a defualt prompt made Optional)
• top_k (Optional with default value as 3 but if number of documents is less than top_k then return as such)

Then we have to run this component and respond with List[Document]. If everything is fine till here I have two questions:

1. Is the example pipeline logic fine or there needs any modifications?
2. Why should we use only ChatGenerator like any advantage? Why not use Generators

utilize our existing ChatGenerators to power the LLM calling

3. How do I handle the response from LLM to generate a List[Document]?

Your thoughts and inputs on the same shall be really helpful. Thanks in advance.

[sjrl]

• User query mandatory

yup!

• Default generator (say OpenAIGenerator i.e. Optional)

Yeah setting a default here makes sense. I'd like to follow the design pattern we have been using elsewhere like the Metadata Extractor. See here

• prompt (with a defualt prompt made Optional)

Yes I think having a default prompt is a good idea. Let's utilize the PromptBuilder under the hood to render the prompt.

• top_k (Optional with default value as 3 but if number of documents is less than top_k then return as such)

I'd set the default to be higher. Probably at least 10.

1. Is the example pipeline logic fine or there needs any modifications?

I still need to have a look at what you provided.

2. Why should we use only ChatGenerator like any advantage? Why not use Generators

You're right we could use either here.

3. How do I handle the response from LLM to generate a List[Document]?

Yeah this is a great question that is still open I would say. Some requirements that I think make sense:

• We should require the LLM to provide a response that is JSON readable. This is straightforward to require with OpenAI (ie. by using response_format) , but unsure of the other LLM providers.
• The response should contain the relevant info to re-sort the documents. So that means we need to be able to map each "score" to the input documents. How does the library you originally linked handle this, is there something there we could utilize here too?

[lbux]

Perhaps it would be better to update the current TransformerSimilarityRanker to support LLM based Rerankers such as https://huggingface.co/BAAI/bge-reranker-v2-gemma. They have code snippers for Reranking based on the Transformers library and their LLM rerankers (Gemma or MiniCPM)

[sjrl]

@lbux that is also interesting! I didn't realize that the LLM rerankers like the one you linked would require a different way to call it. For this I'd actually suggest creating a new issue since I think it is important, but still different from using an LLM like GPT-4o for reranking.