pytest-benchmark
pytest-benchmark copied to clipboard
ionelmc
Sort results by name, save number
Very frustrating in compare when NOW and saved runs are displayed occasionally first or second. Also, on non-compare runs, why one would sort by runtime of unrelated tasks? Sorting by name makes much more sense to me.
I guess you want a different default group-by?
As for the unpredictable printout in compare mode - why is that a problem? If you want to parse it then we should talk about json output or something, otherwise you could sort by name I guess? I could sort by name last but then you get same repr for time but times aren't really equal, so it wouldn't solve anything.
If the current thing were in the same spot each time I will need to just look at first line and see if it's green, now I need to be check each time on what line is NOW. Predictability would simplify iteration when trying different approaches to optimize.
Mostly, the other thing is I need to group by name now, but would prefer to sort by it. Too many empty or techy lines in report when grouping.
Now it's two. Maybe I don't get how this should work, but what's the point in comparing run time of completely different actions?
Basically it's this: comparing different tests allows people to easily compare stuff without doing pytest parametrization. Obviously this don't work for everyone, that's why there are the grouping options.
But why anyone needs to compare different stuff? Isn't a usual use case is to compare same stuff that changes over time?
Not if your comparing different implementations of "something". Parametrization is another way to do it but it can get contrived. Anyway, cat's out of the bag now, can't do much about the feature creep, but we can talk about more sensible defaults :-)
Another thing you could do is assign explicit group name to your tests via marker: http://pytest-benchmark.readthedocs.io/en/stable/usage.html#markers
I second this request, to allow sorting by name (not related to grouping).
Is there an implementation underway? I would be willing to implement it ASAP, and in fact, I more or less need it for development. I am the developer of Construct library. I need to compare "parser" and "compiled parser" of each ~50 classes. Grouping would not work for me.
The benchmark suite: (a 100 tests so far, and more on the way)
@arekbulski what if there was some lightweight group display (no headers, underlines and other noise)?
If you'd sort by name then the slowest/fastest coloring wouldn't group on anything.
I dont think that would work. For example, for "greedybytes", both "parse now" "parse 001" "parse compied now" "parse compiled 001" would need to be in same group. I dont super mind the headers, although dont like them either. This aint possible with grouping, right?
I suppose you could either do @pytest.mark.benchmark(group="something") for every test (cumbersome, yes) or use a pytest hook to automatically mark everything. I'll try to make an example tomorrow (I hope).
I could mark those, its not as cumbersome. But I would also like to see the hook, if possible. Also, I would be willing to implement the name-sorting for myself and others. Thank you for your advice! Also why didnt you just say that sorting by name was already implemented, man!
Here's the example:
import pytest
from pytest_benchmark.fixture import BenchmarkFixture
@pytest.mark.hookwrapper
def pytest_runtest_call(item):
fixture = hasattr(item, "funcargs") and item.funcargs.get("benchmark")
if isinstance(fixture, BenchmarkFixture):
fixture.group = '_'.join(item.name.split('_')[:2])
yield
Alternatively, if you sometimes use the marker (you could check if it's already marked so you don't override a custom group):
import pytest
def pytest_collection_modifyitems(items):
for item in items:
if "benchmark" in getattr(item, "fixturenames", ()):
item.add_marker(pytest.mark.benchmark(group=item.name))