anDoer
Clarification about threshold in keypoint HOTA
In the distance to similarity computation at https://github.com/anDoer/PoseTrack21/blob/35bd7033ec4e1a352ae39b9522df5a683f83781b/eval/posetrack21/posetrack21/trackeval/metrics/hota_pose.py#L37
I'm unable to comprehend the reason to clip the similarity to zero if the (head normalized) distance is greater than 0.5.
This has consequences in (incorrectly) tagging each detection as TP/TN/FP/FN for each alpha ranging from 0.05 to 0.45 at https://github.com/anDoer/PoseTrack21/blob/35bd7033ec4e1a352ae39b9522df5a683f83781b/eval/posetrack21/posetrack21/trackeval/metrics/hota_pose.py#L147
In MOTA calculation, the threshold alpha is fixed at 0.5 i.e., [email protected]. Please refer to https://github.com/leonid-pishchulin/poseval/blob/4258a1575b9f2ddd0bdb85f74557235ab5df0f52/poseval/evaluatePCKh.py#L56.
However, when we try to estimate HOTA at different confidences (0.05 to 0.99), the clipping of similarity has effect of completely neglecting all the [email protected], @[email protected], @0.2, @0.25 @0.3 @0.35 @0.4 aand 0.45.
Futher it additionally calculates [email protected], @0575, @0.625, @0.675 @0.725, @0.775 @0.825 @0.875 @0.925 @0.975 to the original [email protected], @055, @0.6, @0.65 @0.7, @0.75 @0.8 @0.85 @0.9 @0.95
To connect the things into perspective, pascal VOC computes [email protected] whereas COCO computes the [email protected]:0.99:0.05. MOTA calculates [email protected] and hota calculates at all levels @0.05:0.99:0.05.
In short, the distance function may needs to be revisited. I think sim = np.maximum(-1 * distance_mtx + 1, 0) makes it consistent with HOTA metrics based on IoU, MOTA at [email protected].
Please let me know your thoughts and did i missed something?
Thanks for your feedback. I will have a closer look at this problem and I agree that sticking to [email protected]:0.99:0.05 would have been more consistent with existing evaluation metrics.
Hello @anDoer,
A gentle reminder to please check this issue and the next one as well.
Thanks