Mean Average Precision and Pooled Evaluations

[wolfgangkircheis]

I came across an issue with the calculation of the mean average precision of pooled evaluations which results in mAP@k being lower than expected if k is lesser than the number of relevant documents in the pool.

Let r be the number of relevant documents in the pool, P(q,k) the precision of query q at cut-off k, d(q,i) the document at position i in query q and rel(q,d) the relevance of document d for query q with rel: Q × D -> {0,1}.

TrecTools calculates the average precision@k of a query q as

sum({P(q,i) ⋅ rel(q,d(q,i)) | i = {1,...,k}) / r

instead of

sum({P(q,i) ⋅ rel(q,d(q,i)) | i = {1,...,k}) / min(r,k)

resulting in counterintuitive results for AP (and, thus, mAP) if r < k.

I suggest adding a flag to get_map to set the denominator to min(r,k) instead of r for pooled query evaluations.

I've attached a minimal example with a query of ten documents, with the first five of them being relevant, and a total of ten relevant documents in the evaluation pool, for which TrecTools calculates the mAP@5 as 0.5 instead of 1.0.

qrels.txt run.txt

[joaopalotti]

Hi @wolfgangkircheis, I am so sorry for the delay to reply. Thank you very much for using trectools and identifying this bug! I will test it locally as soon as possible, but the solution seems straightforward from your explanation.

Would you be able to create a PR with it, please? Thank you very much in advance!