pose-residual-network icon indicating copy to clipboard operation
pose-residual-network copied to clipboard

Weird evaluation code in evaluate.py

Open jackyjsy opened this issue 5 years ago • 1 comments

Start from Line #140 in evaluation.py, it seems to me that you are using the groundtruth keypoints to obtain you keypoint estimation, which should not happen when you evaluate your PRN network. This issue makes it an improper evaluation.

A brief summary is that, first, you generate indexes from the old_weights_bbox (groundtruth). Then. you seem to utilize the index to place a window around that groundtruth position and calculate your estimated scores. Then the output keypoints are summarized from the scores.

I found the issue in the PyTorch version. There was another guy found the same issue Issue #17. Then I came here and found the same issue in this Keras version. @mkocabas please respond to our concerns. Thanks!

jackyjsy avatar Feb 14 '19 20:02 jackyjsy

@jackyjsy Seems like they're presenting the Both GT part from the Table3 of their work. Here they show how well their PRN network would assign keypoints to person instances if the keypoint and the person segmentation subnet gave perfect outputs.

I feel like @mkocabas and others should have been a bit more explicit with their intent for this repository.

abster95 avatar Mar 08 '19 11:03 abster95