quepid
quepid copied to clipboard
Add option to randomize query results
Is your feature request related to a problem? Please describe. Position bias can influence raters to view higher-ranked items as more relevant.
Describe the solution you'd like I would like an option on the search case to randomize the hits. Randomizing the search results for each query will confound the position bias and lead to better judgements. Maybe this goes into Settings in the Tune Relevance tab.
So, this is going to get done under some sort of notional Human Rater User Experience.....
We now have RANDOM_STRATEGY
for the human rater stuff... So this is done!