quepid icon indicating copy to clipboard operation
quepid copied to clipboard

Add option to randomize query results

Open sstults opened this issue 2 years ago • 1 comments

Is your feature request related to a problem? Please describe. Position bias can influence raters to view higher-ranked items as more relevant.

Describe the solution you'd like I would like an option on the search case to randomize the hits. Randomizing the search results for each query will confound the position bias and lead to better judgements. Maybe this goes into Settings in the Tune Relevance tab.

sstults avatar May 18 '22 14:05 sstults

So, this is going to get done under some sort of notional Human Rater User Experience.....

epugh avatar Aug 03 '22 15:08 epugh

We now have RANDOM_STRATEGY for the human rater stuff... So this is done!

epugh avatar Jan 30 '23 15:01 epugh