How to get the mean Average Performance (mAP) in Segmentation models?

[gcunhase]

1. What is the difference between the following scripts:

python main.py eval
python scripts/evaluate.py --log_file=./mrcnn-dll_evaluate.json
python scripts/inference.py --log_file=./mrcnn-dll_inference.json
python scripts/benchmark_inference.py --batch_size 4 --log_file=./mrcnn-dll_benchmark_inference.json

2. The above scripts return the throughput and latency. How to obtain the accuracy of the model?

[gcunhase]

When running python main.py eval with Mask R-CNN, I get a lot of AP scores. If I had to choose one, which one should I report?

AP: 0.37897980213165283, AP50: 0.5931339263916016, AP75: 0.4121803641319275, APs: 0.23132283985614777, APm: 0.41095027327537537, APl: 0.48839807510375977, ARmax1: 0.3163970410823822, ARmax10: 0.5131967067718506, ARmax100: 0.5427666902542114, ARs: 0.37364697456359863, ARm: 0.5790426731109619, ARl: 0.6804119348526001, mask_AP: 0.3432481288909912, mask_AP50: 0.5592305064201355, mask_AP75: 0.36434662342071533, mask_APs: 0.18999461829662323, mask_APm: 0.3738296627998352, mask_APl: 0.46563971042633057, mask_ARmax1: 0.29616159200668335, mask_ARmax10: 0.46693500876426697, mask_ARmax100: 0.4920860528945923, mask_ARs: 0.3123187720775604, mask_ARm: 0.5282254815101624, mask_ARl: 0.6467187404632568

Edit: this was obtained using the checkpoints from Unet (2D, 3D) and MaskRCNN (TF32, AMP) and giving their path to the script as --model_dir=.../mrcnntf2_tf32_ckpt_21.02.0/.

[jan-golda]

Hi @gcunhase, Sorry for the late response.

The metrics that we report are AP (AP BBox in readme tables) and mask_AP (AP Segm in readme tables). These are the two most important ones. The AP is the average precision of bounding boxes while the mask_AP is for segmentation masks. So if I had to choose one it would be mask_AP.

[jyothir07]

Hi @gcunhase, Sorry for the late response.

The metrics that we report are AP (AP BBox in readme tables) and mask_AP (AP Segm in readme tables). These are the two most important ones. The AP is the average precision of bounding boxes while the mask_AP is for segmentation masks. So if I had to choose one it would be mask_AP.

@jan-golda while evaluation/ inferencing,

predictions = mask_rcnn_model.predict( x=dataset.eval_fn(params.eval_batch_size), callbacks=list(create_callbacks(params))

The predictions['detection_masks'].shape is returning (5000, 100, 28, 28) when evaluating on coco validation data. How is mask_AP calculated from mask resolution of 28x28?

The mAP obtained is Accumulating evaluation results... DONE (t=8.44s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.00000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.00000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.00000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.00000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.00000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.00000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.00000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.00000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.00000 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.00000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.00000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.00000 2021-11-17 12:28:29,881 I dllogger () AP: 0.0, AP50: 0.0, AP75: 0.0, APs: 0.0, APm: 0.0, APl: 0.0, ARmax1: 0.0, ARmax10: 0.0, ARmax100: 0.0, ARs: 0.0, ARm: 0.0, ARl: 0.0, mask_AP: 0.0, mask_AP50: 0.0, mask_AP75: 0.0, mask_APs: 0.0, mask_APm: 0.0, mask_APl: 0.0, mask_ARmax1: 0.0, mask_ARmax10: 0.0, mask_ARmax100: 0.0, mask_ARs: 0.0, mask_ARm: 0.0, mask_ARl: 0.0

[vgthengane]

Can I also integrate the mAP metric to keras.compile() to get mAP while training?