sigsep-mus-eval
sigsep-mus-eval copied to clipboard
sigsep
Move BSS eval to own project?
After the end of SiSEC 2018, this repository might be slightly confusing as it mixes the SiSEC related evaluation with the BSS eval toolkit. I'm pretty sure BSS eval will be further used and developed, so it might be a good idea to create a related BSS eval repository and python package that just contains BSS eval.
yes, this absolutely makes sense. Our plan was to add this to mir_eval (https://github.com/craffel/mir_eval/pull/272). But we also considered a separate repository. You have some opinion on this?
Good point, considering the effort they spent at https://github.com/craffel/mir_eval for transforming v3 into Python it might indeed be the best place to continue the development there. It seems to be also a good idea to document more explicit on the testing and comparison to the first Matlab implementation as the information is a little hidden in different issues at the moment.
A good argument for its own repository might be that the BSS eval community is maybe not too involved in music information retrieval and the other way around, which means it could be a challenging task to maintain https://github.com/craffel/mir_eval as a whole for a long time (e.g. if the interest in the MIR community for it is lost)
- we actually also did thorough comparison of v3 with matlab.
- this v4 implements some new features that make computations way faster (time invariant filters)
I personnaly don't think that having bsseval burried in mireval is much better than having it burried in museval. So yes, I also think a separate repository is the best.
we got more feedback for museval at this years ICASSP. Basically people addressed the need for a separate bsseval package with fewer requirements and focus on performance. Therefore I would now propose to split this project and move the metrics to a new bsseval package. Besides bsseval v3 and v4 we could also think of adding SI-SDR and lightweight implementations of SDR in torch or tensorflow suitable for backprop.
@pseeth @Jonathan-LeRoux would love to also hear your feedback of this
Yeah, happy to contribute. @ethman might have thoughts too. One suggestion is to make any torch/tensorflow dependency optional as those packages can be quite big (won’t fit on an AWS lambda).
One suggestion is to make any torch/tensorflow dependency optional as those packages can be quite big (won’t fit on an AWS lambda).
sure. Lets make separate packages bsseval (==numpy), bsseval-numba, bsseval-pytorch ...
I will start with the basic numpy version and then we could go from there
Having bsseval as a separate package definitely makes sense. I'm also in favor of including SI-SDR / SI-SNR (same thing, different name). I guess we could also consider including the redefined SI-SIR and SI-SAR, although I'm still not convinced they can be reliably interpreted in the way people have, so we may be making the community a service by not providing them :)
I'm also in favor of including SI-SDR / SI-SNR (same thing, different name). I guess we could also consider including the redefined SI-SIR and SI-SAR, although I'm still not convinced they can be reliably interpreted in the way people have, so we may be making the community a service by not providing them :)
:+1: sounds great. @pseeth @Jonathan-LeRoux I will invite you to the repo.