athletic icon indicating copy to clipboard operation
athletic copied to clipboard

Runtime Restriction of Benchmark Methods

Open MatthiasMolitor opened this issue 11 years ago • 3 comments

This change introduces a new (optional) annotation @maxRuntime, which allows one to restrict the runtime of a benchmark method to a specified number of seconds.

This is useful, if a benchmark is executed on different hardware or if a generic event is used to benchmark several implementations with really different performance characteristics. In these cases, you might want to stop a benchmark early instead of waiting for hours until the results of a slow system are available.

In the following example, the benchmark method slowIndexingAlgo() is stopped once 10000 iterations are reached or if the runtime of 5 minutes (300 seconds) is exceeded:

    /**
     * @iterations 10000
     * @maxRuntime 300
     */
    public function slowIndexingAlgo()
    {
        $this->slow->index($this->data);
    }

MatthiasMolitor avatar Jun 21 '14 13:06 MatthiasMolitor

Coverage Status

Coverage increased (+0.77%) when pulling cbd99cad1eeaf5a3eb33f7fea8ccb59784d82f43 on Matthimatiker:restrict_runtime_pull_request into 51fe4b6e5298dd8af187825a4e57745898e37f0e on polyfractal:master.

coveralls avatar Jun 21 '14 13:06 coveralls

@polyfractal Any chance to get this merged?

MatthiasMolitor avatar Sep 04 '15 11:09 MatthiasMolitor

@gabrielsch ^^^^ ?

I've largely stepped aside as maintainer of this library and given it over to @gabrielsch . Unless he has some objections, it LGTM from a casual skim and I'll merge

polyfractal avatar Sep 04 '15 13:09 polyfractal