Fail if benchmark slower than threshold?

[adamalpi]

Is there a way to fail a test if average goes beyond some threshold?

It would be nice to have something like the code below. So it would fail if average time takes more the 0.5 seconds.

@pytest.mark.benchmark(
    max_time_fail=0.5
)
def test_my_stuff(benchmark):
    # benchmark something
    result = benchmark(something)
    assert result == 123

[ionelmc]

You could add something like:

   assert benchmark.stats['min'] < something

[adamalpi]

Hi there, yes, this seems flexible enough. I achieved it via a similar one:

assert benchmark.stats.stats.mean < threshold

Thanks. We can close the issue.

[ssbarnea]

This should be reopened unless is added to the docs example. I was surprised not be able to find a documentation example that would make a test fail if it did run too slow or too fast, this being the main goal when I started looking for a benchmarking solution.

[ionelmc]

Dully noted. This is actually a bigger thing, I'd need to document and explain all the stuff going on in that Benchmark object.

[MarcSkovMadsen]

I was also looking for this in the docs.