feat(prediction): add support of prediction evaluation

[ktro2828]

Signed-off-by: ktro2828 [email protected]

Category

• [x] Perception
  • [x] Prediction

What

Add support of prediction evaluation. Related to https://github.com/tier4/autoware_perception_evaluation/issues/6

Type of change

• [x] New feature
• [ ] Update test
• [x] Update document

Test performed

• sample score results

Frame:
 Total Num: 200
 Skipped Frames: []
 Skipped Frames Count: 0
Ground Truth Num: 13081
ADE: 1.663, FDE: 2.864, Miss Rate: 0.312, Soft mAP: 0.002(Center Distance 3d [m])



Label
car(1.0)
bicycle(1.0)
pedestrian(1.0)
motorbike(1.0)




Predict_num
3771
1189
7526
335


ADE
2.703
nan
0.624
nan


FDE
5.009
nan
0.719
nan


Miss Rate
0.611
nan
0.012
nan


Soft AP
0.000
0.000
0.008
0.000



ADE: 2.247, FDE: 3.198, Miss Rate: 0.260, Soft mAP: 0.056(Center Distance 3d [m])



Label
car(2.0)
bicycle(2.0)
pedestrian(2.0)
motorbike(2.0)




Predict_num
3771
1189
7526
335


ADE
4.049
1.489
1.201
nan


FDE
6.824
1.492
1.278
nan


Miss Rate
0.716
0.000
0.063
nan


Soft AP
0.000
0.181
0.043
0.000



ADE: 1.410, FDE: 2.410, Miss Rate: 0.302, Soft mAP: 0.000(IoU BEV)



Label
car(0.5)
bicycle(0.5)
pedestrian(0.5)
motorbike(0.5)




Predict_num
3771
1189
7526
335


ADE
2.524
nan
0.296
nan


FDE
4.387
nan
0.434
nan


Miss Rate
0.600
nan
0.003
nan


Soft AP
0.000
0.000
0.000
0.000



ADE: 0.255, FDE: 0.427, Miss Rate: 0.000, Soft mAP: 0.000(IoU 3D)



Label
car(0.5)
bicycle(0.5)
pedestrian(0.5)
motorbike(0.5)




Predict_num
3771
1189
7526
335


ADE
nan
nan
0.255
nan


FDE
nan
nan
0.427
nan


Miss Rate
nan
nan
0.000
nan


Soft AP
0.000
0.000
0.000
0.000



ADE: 1.795, FDE: 2.020, Miss Rate: 0.342, Soft mAP: 0.053(Plane Distance [m])



Label
car(2.0)
bicycle(2.0)
pedestrian(2.0)
motorbike(2.0)




Predict_num
3771
1189
7526
335


ADE
2.710
1.477
1.198
nan


FDE
3.300
1.477
1.282
nan


Miss Rate
0.958
0.000
0.069
nan


Soft AP
0.000
0.172
0.042
0.000



ADE: 2.146, FDE: 2.177, Miss Rate: 0.803, Soft mAP: 0.056(Plane Distance [m])



Label
car(3.0)
bicycle(3.0)
pedestrian(3.0)
motorbike(3.0)




Predict_num
3771
1189
7526
335


ADE
2.380
1.886
2.016
2.300


FDE
2.481
1.885
2.041
2.300


Miss Rate
0.994
0.490
0.729
1.000


Soft AP
0.000
0.181
0.044
0.000

Reference

Notes for reviewer