Plausibility checks

[straight-shoota]

I noticed a few issues with the examples shown on this repo:

1. each and each_with_index are both reported to be > 50 times faster than while. Looking at the code it seems like the block might simply be optimized away because the value is not used for anything and the each examples don't iterate at all.

2. reverse_each vs reverse.each looks similar with the former being > 100 times faster.

3. hash vs struct vs namedtuple compares initialization of each container type which is an apples vs. oranges comparison. That doesn't represent any realistic use case.

4. clone vs. dup also compares very different behaviour. Sometimes you need the one, sometimes the other. Choosing between them is usually not based on efficiency but semantics.

5. The results for string concatenation suggest that String#+ is more performant than interpolation. That might be the case when concatenating a few strings, but in general, this is not true.

These are just a few things I noticed immediately. It would probably be a good idea to look more closely at each example and question its plausibility and implications.

[icyleaf]

1. each and each_with_index are both reported to be > 50 times faster than while. Looking at the code it seems like the block might simply be optimized away because the value is not used for anything and the each examples don't iterate at all.
2. reverse_each vs reverse.each looks similar with the former being > 100 times faster.
3. The results for string concatenation suggest that String#+ is more performant than interpolation. That might be the case when concatenating a few strings, but in general, this is not true.

This parts copied from fast-ruby.

3. hash vs struct vs namedtuple compares initialization of each container type which is an apples vs. oranges comparison. That doesn't represent any realistic use case.

Data store uses often in real projects, and developers need know which way store is best, hash and namedtuple is easy solution, but use model by struct is better, right?

4. clone vs. dup also compares very different behaviour. Sometimes you need the one, sometimes the other. Choosing between them is usually not based on efficiency but semantics.

You are right.

BTW, @j8r, @konung and me, we created a new project to initiative to rework https://github.com/language-benchmarks/fast-crystal/pull/1/

[j8r]

There is another issue, you may have an answer @straight-shoota How to have a benchmark result for Array#first and Array#[0] that does't report that one is ~1.10 faster/slower than another? @jhass told me it wasn't really fixable, because the 2 calls are the same. However on the user point a view they aren't.

[icyleaf]

Answered in crystal-lang/crystal#4383 by @RX14.

[j8r]

I understand. What do we do then @icyleaf ? I think for the time being, we can omit them.

[straight-shoota]

Maybe each comparison could be summarised to explain its meaning. And also mention potential issues with each approach.

[j8r]

You're right

require "benchmark"

STR = "abc"
Benchmark.ips do |x|
 x.report "6 * String+" { STR + STR + STR + STR + STR + STR }
 x.report "6 * Interpolation" { "#{STR}#{STR}#{STR}#{STR}#{STR}#{STR}" }
end

Benchmark.ips do |x|                                                                                                                                                          
 x.report "2 * String+" { STR + STR }                                                                                                                                 
 x.report "2 * Interpolation" { "#{STR}#{STR}" }                                                                                                                      
end

      6 * String+   5.17M (193.55ns) (±23.07%)  160B/op   1.16× slower
6 * Interpolation   5.99M (166.93ns) (±15.43%)  240B/op        fastest
      2 * String+  23.37M ( 42.80ns) (±16.89%)  32.0B/op        fastest
2 * Interpolation   8.31M (120.28ns) (±17.93%)   208B/op   2.81× slower

[icyleaf]

Agree with @straight-shoota .