msaf
msaf copied to clipboard
urinieto
Benchmark of algorithms
Hi, do you have any benchmark result of the algorithms? Accuracy(or any other measure), complexity, or whatever. If not, could you recommend me one? I'd like to get segmentations of 10k songs, and I can accept some degradation of performance for significantly low complexity.
Hi, you can refer to my dissertation for some detailed scores (especially in the Appendices) for the existing algorithms in MSAF.
Generally, I would recommend Structural Features (sf) for boundaries, and 2D-Fourier Magnitude Coefficients (fmc2d) for labels.
Regardless, convolutional neural networks should outperform any of the existing algorithms in MSAF right now (this is the latest publication about it, I believe). I might include an implementation in MSAF sometime in the future (right, @f0k? :)).
I might include an implementation in MSAF sometime in the future
Yeah, thanks for the reminder! I'll still have to figure out what's possible. Maybe the most realistic setting would be us posting the implementation from MIREX 2014 under a license not for commercial use. That would mean it would live outside of MSAF, but it could produce outputs in a compatible format so you can still use MSAF for labelling -- I think I remember it was relatively straightforward to use MSAF to label existing segmentations? Alternatively, MSAF could probably import and use it.
I can accept some degradation of performance for significantly low complexity.
Our CNNs are not tuned towards low complexity -- if you need high throughput, a simple hand-crafted algorithm may be better. A complex hand-crafted algorithm will probably be slower than the CNNs. Would be interesting to do a direct comparison some time.
On MacPro 2014 (no multi processing) with 10 wave files. CNMF took 1555 seconds and Spectral clustering: 211 seconds. Posting for someone who might be also interested, performance was not evaluated though.
Thanks! Yeah, C-NMF can be demanding. Probably the Checkerboard kernel (foote) is the fastest at finding boundaries. And 2D-FMC (fmc2d) the fastest at labeling the segments.