Parallel execution makes benchmark unfair

[samdark]

It seems parallel execution of benchmark makes it unfair. It depends on execution order very much and one test affects another.

In order to make it fair all parsers should be tested sequentionally. PHP process should be killed after each test.

[hkdobrev]

It depends on execution order very much and one test affects another.

@samdark Why is that?

As a side note: Don't you think every iteration should be run in a separate process? This way PHP optimizing some regex, because of the multiple execution would not be a factor.

[cebe]

@hkdobrev if the test machine has enough cores to have one free for each parser and also other for other processes running on the system and the scheduler of the operating system will allow a process to run on its core without beeing interrupted it would be fair running in parallel. As long as parser processes need to "fight" for resources one might get more time than the other while running the benchmark.

Seen very different results on different runs on my machine already.

[hkdobrev]

@cebe Thanks for the explanation!

I agree, everything should be run sequentially. But I think it is very important every driver (parser dialect) and every profile to be run in separate PHP processes. So nothing is reused especially regex compilations and optimizations.

It is possible several parsers to reuse the same regex pattern. But even in a single driver if you run the default profile first and then run the GFM profile, PHP could use some regex pre-compilations or optimizations made during the execution of the first profile.

This is related, because AFAIK Parallel.php is starting new system processes for each task. So it achieved this task by parallelization.

We could achieve correct sequential benchmarking by running a new blocking PHP process for every driver and every profile.