sigsep-mus-eval icon indicating copy to clipboard operation
sigsep-mus-eval copied to clipboard

Allow silent targets and estimates

Open DavidDiazGuerra opened this issue 1 year ago • 3 comments

Hello,

I'm proposing this PR following the discussions of #94. With this implementation, mus-eval raises a warning instead of an error if one of the sources or estimates is completely silent and compute_permutation is False (if it is True it raises the same error it used to).

Some important notes:

  • It handles silent sources or estimates in a way equivalent to how the library already handled silent frames: the metric is set to NaN or -Inf in that song and then the song is excluded when computing the average metric of the dataset.
  • This does not modify any of the results that could be computed before, it just allows getting results for inputs that raised errors.

I think this is important for dealing with datasets more complex than MusDB18, where a wider variety of instruments could be available in the whole dataset but not all the instruments might be present in every song (this situation could be quite common, for example, in classical music).

Best, David

DavidDiazGuerra avatar Nov 09 '23 09:11 DavidDiazGuerra

Sorry for the bug in the previous version. Now the only reason the tests are not passed is that test_silent_input is obviously not raising a ValueError. I'm not sure if you would prefer to modify or just remove that test.

DavidDiazGuerra avatar Nov 09 '23 10:11 DavidDiazGuerra

@DavidDiazGuerra maybe modify the tests

faroit avatar Jan 16 '24 18:01 faroit

Done!

DavidDiazGuerra avatar Jan 26 '24 08:01 DavidDiazGuerra