quepid
quepid copied to clipboard
Add MRR (mean reciprocal rank) as a scorer
Is your feature request related to a problem? Please describe. MRR is a handy metric for evaluating known or single item search.
Describe the solution you'd like implement a new scorer to compute this metric. Related to #521 and #522 if implemented the associated documentation issues need to be addressed.
Describe alternatives you've considered none.
Additional context none.
ERR with binary judgments is MRR, so implement ERR and use it to compute MRR
I'm not sure I'd tie the two together in the implementation. What if we decided to remove ERR, maybe because hardly anyone uses it? The code would also be a lot easier to understand if someone is just interested in MRR, if it wasn't handled as a special case of the more complex ERR.
I would imagine known item searches with multi-graded relevance values would benefit from having ERR. There is no particular reason to implement it concurrent with RR, I just prefer maintaining one block of code over two... Given that RR is trivial, it isn't likely to need much maintenance.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Please do advocate for this issue and ideally submit a patch to get the attention of the maintainers!