Runtime Restriction of Benchmark Methods
This change introduces a new (optional) annotation @maxRuntime, which allows one to restrict the runtime of a benchmark method to a specified number of seconds.
This is useful, if a benchmark is executed on different hardware or if a generic event is used to benchmark several implementations with really different performance characteristics. In these cases, you might want to stop a benchmark early instead of waiting for hours until the results of a slow system are available.
In the following example, the benchmark method slowIndexingAlgo() is stopped once 10000 iterations are reached or if the runtime of 5 minutes (300 seconds) is exceeded:
/**
* @iterations 10000
* @maxRuntime 300
*/
public function slowIndexingAlgo()
{
$this->slow->index($this->data);
}
Coverage increased (+0.77%) when pulling cbd99cad1eeaf5a3eb33f7fea8ccb59784d82f43 on Matthimatiker:restrict_runtime_pull_request into 51fe4b6e5298dd8af187825a4e57745898e37f0e on polyfractal:master.
@polyfractal Any chance to get this merged?
@gabrielsch ^^^^ ?
I've largely stepped aside as maintainer of this library and given it over to @gabrielsch . Unless he has some objections, it LGTM from a casual skim and I'll merge