spotlight
spotlight copied to clipboard
maciejkula
MRR score uses a "mean" instead of "min"
If I'm not mistaken, MRR score is a mean of reciprocal rank scores over a set of examples, where reciprocal rank is given as 1 / (rank of the first positive item).
Currently implementation in spotlight takes a mean over ranks of positive items. I think this is not correct. It is also inconsistent with lightfm. It also makes it impossible to have mrr score of 1 even with perfect model (even if positive items are ranked [1, 2, 3], taking the mean gives result lower than 1).
Can I send a PR to fix this?
elanmart, I think worst case you can make a fix on repository you forked and share it here, I would also love to see it
It's not necessarily incorrect, but I can see why you might want to do it differently.
Could you make a PR that keeps the current code as default, but allows the alternative as an option? You could pass a mode argument that's either mean or min.
Can I bump this. I really think that all literature discusses reciprocal rank as a per-user measure (per-query for information retrieval), and the average is taken across users. For instance, in the Croft book page 319, "reciprocal rank... is defined as the reciprocal of the rank at which the first relevant document is retrieved"
Therefore .min() should be the default.
Taking the .mean() is more similar of the reciprocal ranks is, in an adhoc way, to Mean Average Precision, and should be deprecated in favour of that.
Happy to contribute a PR. My own implementation also supports rank cutoffs and not dropping users with no relevant test items, but setting them to 0 in the array.
