torchmetrics

torchmetrics copied to clipboard

🐛 Bug

Training a model with detectron2 and evaluating it with the built-in cocoeval results in different mAP estimates for both box and segm metric cases.

To Reproduce

1. Train model with detectron2
2. Use CocoEval to generate mAP, mAR, etc
3. Use torchmetrics to calculate mAP, mAR, etc
4. Compare for a detection and segmentation models

Expected behavior

The values generated to be the same given both use underlying pycocotools (at least for segm). Or up to some floating point precision.

See below for box case:

metricName	torchmetrics	Detectron2 Cocoeval (x100)
map	0.6738	66.01083445431875
map_50	0.9600	97.36921342328598
map_75	0.8268	83.13779622663151
map_small	0.5943	64.11872772143319
map_medium	0.6192	64.02383668112452
map_large	0.8431	77.85958548685628
mar_1	0.7376
mar_10	0.7394
mar_100	0.7394
mar_small	0.6466
mar_medium	0.7293
mar_large	0.8795
map_per_class	0.6738	NaN, NaN, NaN, NaN, 66.01083
mar_100_per_class	0.7394

Environment

• TorchMetrics version 9.3 (pip)

Hi! thanks for your contribution!, great first issue!

I have also encountered the same issue. Is that bug fix in the agenda?