Accuracy of person-detection-retail-0013 is much lower than the reference on COCO dataset

[duanyao]

I run accuracy_checker with person-detection-retail-0013/FP32 model on coco val2017 dataset, and the mAP is 47.62%, much lower than the reference(0.8862). What might be wrong?

The command and result:

(v39) duanyao@duanyao-laptop-c:~/project/openvino_model_zoo.git$ python tools/accuracy_checker/ac_main.py -td CPU -c tools/accuracy_checker/configs/person-detection-retail-0013.yml -s ~/project/ai-dataset/COCO/ -m intel/person-detection-retail-0013/FP32/ -d tools/accuracy_checker/dataset_definitions.yml  --stored_predictions /home/duanyao/project/annotation/detection_results/coco/person-detection-retail-0013-fp32/det5/det5.1.pickle
2021-11-08 14:51:03.447484: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Processing info:
model: person-detection-retail-0013
launcher: dlsdk
device: CPU
dataset: person_detection
OpenCV version: 4.5.4-dev
Annotation conversion for person_detection dataset has been started
Parameters to be used for conversion:
converter: mscoco_detection
annotation_file: /home/duanyao/project/ai-dataset/COCO/annotations/person_keypoints_val2017.json
has_background: True
sort_annotations: True
use_full_label_map: True
Total annotations size: 5000
100 / 5000 processed in 0.283s
200 / 5000 processed in 0.316s
300 / 5000 processed in 0.307s
400 / 5000 processed in 0.266s
500 / 5000 processed in 0.282s
600 / 5000 processed in 0.266s
700 / 5000 processed in 0.277s
800 / 5000 processed in 0.262s
900 / 5000 processed in 0.337s
1000 / 5000 processed in 0.305s
1100 / 5000 processed in 0.271s
1200 / 5000 processed in 0.259s
1300 / 5000 processed in 0.283s
1400 / 5000 processed in 0.258s
1500 / 5000 processed in 0.272s
1600 / 5000 processed in 0.275s
1700 / 5000 processed in 0.323s
1800 / 5000 processed in 0.274s
1900 / 5000 processed in 0.264s
2000 / 5000 processed in 0.281s
2100 / 5000 processed in 0.255s
2200 / 5000 processed in 0.284s
2300 / 5000 processed in 0.284s
2400 / 5000 processed in 0.343s
2500 / 5000 processed in 0.261s
2600 / 5000 processed in 0.291s
2700 / 5000 processed in 0.270s
2800 / 5000 processed in 0.296s
2900 / 5000 processed in 0.259s
3000 / 5000 processed in 0.326s
3100 / 5000 processed in 0.316s
3200 / 5000 processed in 0.280s
3300 / 5000 processed in 0.266s
3400 / 5000 processed in 0.270s
3500 / 5000 processed in 0.267s
3600 / 5000 processed in 0.270s
3700 / 5000 processed in 0.300s
3800 / 5000 processed in 0.344s
3900 / 5000 processed in 0.274s
4000 / 5000 processed in 0.262s
4100 / 5000 processed in 0.273s
4200 / 5000 processed in 0.249s
4300 / 5000 processed in 0.271s
4400 / 5000 processed in 0.264s
4500 / 5000 processed in 0.336s
4600 / 5000 processed in 0.269s
4700 / 5000 processed in 0.289s
4800 / 5000 processed in 0.263s
4900 / 5000 processed in 0.274s
5000 / 5000 processed in 0.264s
5000 objects processed in 14.132 seconds
Annotation conversion for person_detection dataset has been finished
person_detection dataset metadata will be saved to mscoco_person_detection.json
Converted annotation for person_detection dataset will be saved to mscoco_person_detection.pickle
IE version: 2021.4.0-3839-cd81789d294-releases/2021/4
Loaded CPU plugin version:
    CPU - MKLDNNPlugin: 2.1.2021.4.0-3839-cd81789d294-releases/2021/4
Found model intel/person-detection-retail-0013/FP32/person-detection-retail-0013.xml
Found weights intel/person-detection-retail-0013/FP32/person-detection-retail-0013.bin
Input info:
        Layer name: data
        precision: FP32
        shape: [1, 3, 320, 544]

Output info
        Layer name: detection_out
        precision: FP32
        shape: [1, 1, 200, 7]

5000 objects processed in 327.106 seconds                                                                                                                                 
prediction objects are save to /home/duanyao/project/annotation/detection_results/coco/person-detection-retail-0013-fp32/det5/det5.1.pickle
map: 47.62% [FAILED:  abs error = 41.0 | relative error = 0.4627]

Pickle files produced by accuracy_checker is here:

outputs.zip

Software version used:

python 3.9.2 openvino 2021.4.0 (installed by pip) accuracy-checker 0.9.0 (git version 661d740ed ("debug pickle loading (#2852)", 2021-11-01) )

ac_main.py:

from openvino.tools.accuracy_checker.main import main
if __name__ == '__main__':
    main()

[eaidova]

@duanyao model accuracy reference provided on internal testset, not COCO dataset. Unfortunatly, we can not share details about our private data

[duanyao]

I see. Could you also provide an accuracy reference on any public dataset?