Unexpected benchmark run order

[pmdartus]

Observations

I realized that my mental model with how the tests are executed by best doesn't match with the way the framework actually run.

benchmark('A', () => {
  console.log('benchmark A');

  before(() => console.log('benchmark A - before'));
  run(() => console.log('benchmark A - run'));
});

benchmark('B', () => {
  console.log('benchmark B');

  before(() => console.log('benchmark B - before'));
  run(() => console.log('benchmark B - run'));
});

My mental model was that the console would print in the following order:

benchmark A
benchmark B

N times:
  - benchmark A - before
  - benchmark A - run

N times:
  - benchmark B - before
  - benchmark B - run

While in reality the messages prints in the following order:

benchmark A
benchmark B

N times:
 - benchmark A - before
 - benchmark A - run
 - benchmark B - before
 - benchmark B - run

One of the side effects with the current ordering is that by alternating between the different benchmark, one of the benchmark performance may be impacted by another benchmark. For example, if benchmark A allocates a lot of short-living objects it may cause the benchmark B to stop longer due to unexpected GC.

In the case where each benchmark independently, there would be a lower chance to have side effects between benchmarks.

More details: here

Versions

• node: 10.16.0
• best: 4.0.0-alpha4