scikit-learn_bench icon indicating copy to clipboard operation
scikit-learn_bench copied to clipboard

reporting format of benchmarks

Open amueller opened this issue 6 years ago • 0 comments

As discussed here: https://github.com/scikit-learn/scikit-learn/pull/14247#issuecomment-508166336

I think the current report is very hard to read. It might be helpful to specify very clearly what the baseline is, that is the meaning of 1 in all the plots - it's your own C++ implementation.

For a comparison with scikit-learn I think doing sklearn speed / your c++ speed would be easier to read as it shows your speedup factor, not our slow-down factor.

Finally, I don't see the number of cores in your benchmark, which is pretty crucial since most of our implementations are single-threaded. Yes, that's a big issue, but saying "we're 100x faster" without saying "on 100 CPUs instead of 1" is quite misleading. It might be helpful to have a chart of speedup vs number of CPUs.

amueller avatar Jul 05 '19 15:07 amueller