Return the first, instead of the last result in benchmark fixture

[jvesely]

The current behaviour is to return the last result of multiple iterations. This makes testing stateful functions difficult. For example:

import pytest

class foo():
    def __init__(self):
        self.x = 0

    def inc(self):
        self.x = self.x + 1
        return self.x


@pytest.mark.benchmark
def test_inc(benchmark):
    f = foo()

    res = benchmark(f.inc)
    assert res == 1

The above snippet only passes with --benchmark-disable and fails with benchmarks enabled. I know it's possible to get the number of executions from the benchmark object, but the expected result might not be as easy to calculate as above.

one alternative is:

res = f.inc()
assert res == 1
if (benchmark.enabled):
   benchmark(f.inc)

but that complains/warns that the benchmark fixture is unused, so the correct workaround would be

res = f.inc()
assert res == 1
if (benchmark.enabled):
   benchmark(f.inc)
else:
   benchmark(lambda :None)

[Jwely]

I encountered this problem when benchmarking a function which mutates its own input arguments. The first run, this mutation had a side effect, but on subsequent runs, the mutation wasn't necessary, and so the side effect was not observed.

Workaround is to encpasulate this function in a wrapper that deepcopies any initial state and provides the copy to the stateful function to mutate.

use the benchmark_safe wrapper on the benchmark call instead.