neural-search
neural-search copied to clipboard
[FEATURE] Add Rank based combination technique for Hybrid Search
Is your feature request related to a problem?
Hybrid search's current normalization techniques considers scores to bring different sub queries to same scale. Using this approach, the top-ranked result will invariably receive a score of 1, while the lowest-ranked result will consistently be assigned a score of 0, regardless of their proximity or distance from the search query. Integrating the normalized scores with additional sub-queries can potentially introduce bias, as the score distribution may influence the overall results for certain data sets.
What solution would you like?
As an alternative normalization and combination technique, Reciprocal Rank Fusion (RRF) for Hybrid Search utilizes the positions or ranks of the documents rather than relying solely on their scores. This approach aims to mitigate potential biases that may arise from combining scaled scores with other sub-queries.