haystack icon indicating copy to clipboard operation
haystack copied to clipboard

feat: Add a calibration factor to `TransformersSimilarityRanker` to increase spread of scores

Open sjrl opened this issue 1 year ago • 1 comments

Is your feature request related to a problem? Please describe. I would like to add a calibration_factor similar as to the ExtractiveReader https://github.com/deepset-ai/haystack/blob/5ac56ebdaf18c6f1fefb63098c69a96b579181eb/haystack/components/readers/extractive.py#L66

to the TransformersSimilarityRanker to similarly calibrate the final scores (aka probabilities) output by the ranker. This could be helpful because we find in practice a lot of Cross-Encoder models output scores in a very narrow range (e.g. between 0.3-0.5) instead of utilizing the full 0 to 1 range of the underlying sigmoid function.

This could be helpful in combination with other components like TopPSampler which rely on there being a larger spread in scores.

sjrl avatar Aug 14 '24 07:08 sjrl

can I get this issue?

MetroCat69 avatar Aug 22 '24 07:08 MetroCat69