benchmark fixture runs code an additional 2 times

[phdoerfler]

I noticed that whenever I put the benchmark() fixture around my code, it runs 2 additional times. I noticed because this code runs from multiple seconds to multiple minutes per invocation. This really adds up quickly.

Details

meep = 0

@pytest.mark.parametrize("sig", [sig.CPignificance])
@pytest.mark.parametrize("img_name", ['camera'])
@pytest.mark.parametrize("size", [2**n for n in range(8, 9)])
def test_large(sig, img_name, size, benchmark):
    def encode_image(img):
        global meep
        ed = SpihtImage(img, significance_strategy = sig)
        meep = meep + 1
        print("This is run #%d" % meep)
        res = ed.encoded
        print("Thanks for the fish #%d" % meep)
        return ed, res

    img = SpihtTest().image(size = size, name = img_name)
    print("Before the benchmark")
    ed, _ = benchmark(encode_image, img)
    # ed, _ = encode_image(img)
    print("Benchmark done")
    SpihtTest().assertNoDifference(ed, "difference using %s at %dx%d" % (sig, size, size))

I'm using pytest-3.1.1 (anaconda):

conda list pytest-benchmark
# packages in environment at /usr/local/miniconda3/envs/spiht:
#
# Name                    Version                   Build  Channel
pytest-benchmark          3.1.1                    py36_1

I invoke pytest like this:

pytest src/test/python/benchmark.py --benchmark-min-rounds=1 \
--benchmark-group-by=param:sig,param:img_name,name --benchmark-cprofile=tottime "$@"

[ionelmc]

@phdoerfler so if you have very slow test maybe pedantic mode is more suited for you: https://pytest-benchmark.readthedocs.io/en/latest/pedantic.html