kge
kge copied to clipboard
uma-pi1
Report Adjusted Mean Rank (AMR)
The the adjusted mean rank (AMR) complements the reported MRR and hits@k metrics - it has been demonstrated to provide more fair comparisons between datasets and varying testing set sizes. It would be great if LibKGE could report this metric in its ensemble.
Citation:
Max Berrendorf, Evgeniy Faerman, Laurent Vermue, and Volker Tresp (2020) Interpretable and Fair Comparison of Link Prediction or Entity Alignment Methods with Adjusted Mean Rank. arXiv, 2002.06914.
I saw that paper a while ago and I really like it (esp. also the Appendix). I think more metrics are welcome. Could you provide a PR for this feature?
I did a bit of digging in the code to see where this might fit. I'm not sure it will be so easy for me to make a contribution
The AMR is a rescaling of the mean rank by a test set-dependent constant (so, nothing will change qualitatively for a fixed test set). To report it, we'd need to compute the average number of non-filtered candidates during evaluation, and then divide the mean rank by half of that. This probably should be done by extending
https://github.com/uma-pi1/kge/blob/e7bd9b7c131452677343ca4989fa2cc8b63c3111/kge/job/entity_ranking.py#L477
to also report the number of filtered subjects and objects. From this, we can also compute the adjusted mean rank index.
