Incorrect edge case

[cannon]

This library gives incorrect results in a specific edge case. In the MOT16 paper 4.1.1 it states:

if a ground truth object i is matched to hypothesis j at time t − 1 and the distance (or dissimilarity) between i and j in frame t is below td, then the correspondence between i and j is carried over to frame t even if there exists another hypothesis that is closer to the actual target.

However, in this library it checks whatever the most recently matched hypothesis j was, even if that match did not occur in time t-1. This creates an incorrect metric when:

At time 1, object A matches prediction X At time 2, prediction X moves too far away from object A, and no other prediction matches it so it is a false negative. At time 3, prediction X moves back to being close enough to match object A, but prediction Y appears and is even closer to object A.

In the authors' metric (https://bitbucket.org/amilan/motchallenge-devkit/src/default/utils/clearMOTMex.cpp) this results in object A matching to prediction Y, however this library will match it to prediction X.

Another issue I found that is that num_frames does not count frames with zero objects and zero predictions. Any metrics down the line which divide by num_frames would be affected.

[cheind]

Thanks for reporting. I didn't have time yet to look into the issues you mention. The first one is surely bug, the second one I'm unsure about. If neither an object nor an hypothesis is present I wonder why we should count that as a frame.

[cannon]

In your ground truth data you would probably have some frames with no objects, and you want to test that the predictor doesn't find any objects that don't exist. If it did, that would give you a false positive but the frame would still be counted. If it didn't find any false positive, the frame count won't increment.

What i'm not sure about is if there are any other metrics which use num_frames. Maybe the average number of errors per frame or something (???). If a metric like this did exist its value would be wrong because num_frames is too low.

The num_frames issue isn't a big deal if you prefer how it is. I think the edge case really should be fixed though. Basically the issue there is MOTAccumulator's self.m attribute maintains associations from more than one frame ago.

[Shirly-Perl]

I'm joining the question about the method num_frames. It yields lower result when frame contains no objects. The problem is in the method self.update in MOTAccumulator - it does not handle the case where no=nh=0. It can affect a manual calculation of FA-rate (FA per frame), but it's not in use by the program.

[cheind]

@cannon yes you are right, that's an oversight.