WasmJs: in a batch consisting of multiple benchmark functions the first one is always runs faster

[fzhinkin]

trafficstars

While analyzing the performance of some Kotlin stdlib functions, I faced the following issue: when running wasmJs benchmarks, the first benchmark among the batch always runs faster.

Consider the following example:

@Benchmark fun a(blackhole: Blackhole) { blackhole.consume(1) }
@Benchmark fun b(blackhole: Blackhole) { blackhole.consume(2) }
@Benchmark fun c(blackhole: Blackhole) { blackhole.consume(3) }

All three functions are identical (module the numeric literal), but when benchmarked, a is always faster:

Benchmark                 Mode  Cnt  Score   Error  Units
BenchmarkWithBlackhole.a  avgt    5  4.961 ± 0.006  ns/op
BenchmarkWithBlackhole.b  avgt    5  9.036 ± 0.023  ns/op
BenchmarkWithBlackhole.c  avgt    5  9.034 ± 0.022  ns/op

If the first function is excluded and now the run consists of only two benchmarks, the one executed earlier will be faster again:

Benchmark                 Mode  Cnt   Score   Error  Units
BenchmarkWithBlackhole.b  avgt    5   5.250 ± 0.032  ns/op
BenchmarkWithBlackhole.c  avgt    5  11.328 ± 0.051  ns/op

The issue is not reproducible with js target.

You can find a demo project here: https://github.com/fzhinkin/kotlinx-benchmark-wasmjs-issue

[fzhinkin]

I tried to analyze V8 logs (I have no expertise in that area), but didn't find anything that could explain performance degradation after the first benchmark execution finished.

[fzhinkin]

BTW, it seems like the regular JS target is also affected when BuiltIn executor is used instead of BenchmarkJs