benchmark icon indicating copy to clipboard operation
benchmark copied to clipboard

Make benchmarks nicer to read by default, via the commandline.

Open rubyFeedback opened this issue 7 months ago • 0 comments

The ruby benchmark ecosystem is quite nice; it reports things, aka what is faster aka more efficient.

Take the following old code I have:

# Interpolation is slowest, interestingly enough.
require 'benchmark'

N_TIMES = 5_000_000

a = 'foo'.freeze
b = 'bar'.freeze

Benchmark.bmbm(30) { |entry|
  entry.report('(1) interpol: #{a}#{b}  ') { N_TIMES.times { "#{a}#{b}" } }
  entry.report('(2) plus:     a+b       ') { N_TIMES.times { a + b      } }
  entry.report('(3) append:   a.dup << b') { N_TIMES.times { a.dup << b } }
}

Results are:

Rehearsal ------------------------------------------------------------------
(1) interpol: #{a}#{b}           0.534171   0.005792   0.539963 (  0.539974)
(2) plus:     a+b                0.419223   0.000000   0.419223 (  0.419227)
(3) append:   a.dup << b         0.413437   0.003269   0.416706 (  0.416715)
--------------------------------------------------------- total: 1.375892sec

                                     user     system      total        real
(1) interpol: #{a}#{b}           0.544134   0.006647   0.550781 (  0.550799)
(2) plus:     a+b                0.417118   0.000000   0.417118 (  0.417120)
(3) append:   a.dup << b         0.421809   0.013338   0.435147 (  0.435174)

Now from these results, it seems interpolation is slower than the other two methods. I take it the most important is the very right value in () parens.

But I am not sure if I interprete this correctly.

Would it be possible for ruby to add a "verbose" output result, aka a human-readable summary? With that I mean something like "did you mean" gem, but adjusted to benchmark interpretation.

Something like:

"The above results indicate that interpolation (1) is fastest, whereas the other two methods are -15% (2) and -14% (3) slower."

The above sentence should just indicate what I am trying to convey.

I'd like for benchmark to become more useful to normal people without a PhD. Just like the did-you-mean gem, which is super-simple and suggests what could be wrong. Sometimes this is not quite correct, but most of the time did-you-mean is super-correct and easy to understand. Benchmark evaluation are not so easy to understand and I'd like a dedicated "report" subsystem there, that is human-readable. Aka the output should be simpler, and using that subsystem in the benchmarks module should ALSO be super-simple, even the default; or, if not the default, then a simple option, a simple API, to call it, such as:

Benchmark.bmbm(30, :be_verbose) { |entry|

Or API-wise:

Benchmark.verbose_bmbm(30) { |entry|
Benchmark.bmbm_verbose(30) { |entry|

Or some other, somewhat similar name.

If added then this should also be documented, at the least with one usage example, so people can quickly start incorporating this into their own benchmarks.

rubyFeedback avatar Jul 02 '24 11:07 rubyFeedback