mir_eval icon indicating copy to clipboard operation
mir_eval copied to clipboard

Numerical stability issues in floating point comparisons (chord directional hamming)

Open bmcfee opened this issue 5 years ago • 7 comments

I ran into a funny bug just now, and wanted to get some opinions on whether this should be considered an mir_eval problem or not.

This check in the directional hamming distance function looks for disjoint time intervals by comparing the end of one interval to the start of the next.

This is fine in principle, but I have a few (reference) files which are stored in JSON (jams) format, and for whatever reason, deserializing them messes up the floating point precision just right so that it triggers an error. For example:

In [4]: J = jams.load('/home/bmcfee/data/crema/chords/TRAIIEF149E3861F6C.jams')                                                                                                                                    

In [5]: ints, labs = J.annotations[0].to_interval_values()                                                                                                                                                         

In [6]: ints[1, 1]                                                                                                                                                                                                 
Out[6]: 2.126712

In [7]: ints[2, 0]                                                                                                                                                                                                 
Out[7]: 2.1267119990000003

On the one hand, this is clearly a data error, and it's reasonable for mir_eval to fail here.

On the other hand, this is not a serious data error, and it's one that we can expect to pop up any time we're dealing with numerical data through text serialization, which is basically every use case of mir_eval.

Given that, what do people think of rounding the time intervals (say to the millisecond level) prior to checking? That is, just add lines to the effect of

reference_intervals = np.around(reference_intervals, decimals=3)

before the overlap check. For my money, I see no downside to this, but I'd like to get some opinions before opening a PR.

bmcfee avatar Dec 03 '18 13:12 bmcfee