kotlinx-benchmark icon indicating copy to clipboard operation
kotlinx-benchmark copied to clipboard

Consider unifying messagesg in kotlinx-benchmark executors with JMH's ones

Open CommanderTvis opened this issue 2 years ago • 0 comments

  1. Messages printed by JMH are much more meaningful.
  2. Making messages formats similar improves the library user's experience.
Consider similar benchmark reports. The JMH one:
… space.kscience.kmath.benchmarks.ExpressionsInterpretersBenchmark.mstExpression

Warm-up 1: 2.464 ops/s
Warm-up 2: 2.530 ops/s
Iteration 1: 2.575 ops/s
Iteration 2: 2.561 ops/s
Iteration 3: 2.525 ops/s
Iteration 4: 2.509 ops/s
Iteration 5: 2.517 ops/s
Iteration 6: 2.493 ops/s
Iteration 7: 2.506 ops/s
Iteration 8: 2.477 ops/s
Iteration 9: 2.548 ops/s
Iteration 10: 2.549 ops/s

  Success: 2.526 ±(99.9%) 0.047 ops/s [Average]
  (min, avg, max) = (2.477, 2.526, 2.575), stdev = 0.031
  CI (99.9%): [2.479, 2.573] (assumes normal distribution)

The one by JsExecutor:

… space.kscience.kmath.benchmarks.ExpressionsInterpretersBenchmark.mstExpression
Iteration #0: 0.0569060 ops/sec
Iteration #1: 0.0558908 ops/sec
Iteration #2: 0.0571284 ops/sec
Iteration #3: 0.0568593 ops/sec
Iteration #4: 0.0571386 ops/sec
Iteration #5: 0.0551405 ops/sec
Iteration #6: 0.0564571 ops/sec
Iteration #7: 0.0558157 ops/sec
Iteration #8: 0.0585252 ops/sec
Iteration #9: 0.0602421 ops/sec
  Success:   ~ 0.0570104 ops/sec ±1.6%

It is obvious that JMH's format contains more information: warmup and measurement iterations are separated, and there are basic statistics of results.

CommanderTvis avatar Jan 27 '22 20:01 CommanderTvis