Fixing error metrics as discussed in #27

[Selmaan]

I implemented these changes, plus two minor additions.

First, for keypoints which are supposed to be not-visible, it seemed reasonable to replace their confience score with 1 minus the confidence score, for the purposes of reporting online during training, since the target confidence for these points is 0. Euclidean errors however should be masked out, since they are not defined for non-visible keypoints. These changes only affect values printed to command line during training, but not logged values to disk. This is fine, because the keypoint errors logged to disk includes NaN values for non-visible keypoints

Second, in debugging these changes, I noticed that the logger callback appeared to be broken. Instead of adding a new entry for each epoch, it appears to have a first entry with all zeros, and a second entry with only the most recent values. This looked like a single line got misplaced outside a conditional statement, and I tested that logger works correctly after making this change.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[Selmaan]

This push is to fix the issues discussed in open issue #27

[jgraving]

I'd get rid of all the code for detecting nans and just use np.nanpercentile and np.nanmean. That would greatly simplify things and give the same result. If you could update to use that then I'll merge.

[Selmaan]

We do have to detect nans still, at least to do what I've implemented here, because the confidence values for non-visible target keypoints should be 0. But I can make the other changes you mention, and sorry about the commented code...that's deleted in my version of the code I must have pushed the wrong version.

The code is a bit complicated because I didn't want to remove the option to do confidence thresholding that you had already built in. If it's ok to get rid of that, the code is much simpler.

On Sun, Apr 26, 2020 at 4:45 AM Jake Graving [email protected] wrote:

@jgraving commented on this pull request.

In deepposekit/callbacks.py:

•    euclidean_perc, confidence_perc = keypoint_percentile
  

•    euclidean_mean, confidence_mean = np.mean([euclidean, confidence], axis=1)
  

•        y_visible = y_visible[mask]
  

•    if ~y_visible.all():
  

•        euclidean = euclidean[y_visible]
  

•        confidence[~y_visible] = 1-confidence[~y_visible]
  

•    euclidean_perc = np.percentile(euclidean, [0, 5, 25, 50, 75, 95, 100])
  

•    euclidean_mean = euclidean.mean()
  

•    confidence_perc = np.percentile(confidence, [0, 5, 25, 50, 75, 95, 100])
  

•    confidence_mean = confidence.mean()
  

•    # keypoint_percentile = np.percentile(
  

•    #     [euclidean, confidence], [0, 5, 25, 50, 75, 95, 100], axis=1
  

•    # ).T
  

Just delete this commented code. git keeps track of changes. There's no need to keep old code.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jgraving/DeepPoseKit/pull/50#pullrequestreview-400477025, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABJNQGS56CNHGNTZGUL747TROPYAVANCNFSM4MLRFRPA .

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity.