More detailed params info

[antocuni]

Currently for each benchmark we save also the "param" attribute, which is directly taken from pytest's callspec.id. This works very well if you have only one parametric fixture, but it becomes less useful if you have more. For example, consider this real life example: https://bitbucket.org/antocuni/capnpy/src/22749ed7be02fc969907391e38cac1fe4f925e18/capnpy/benchmarks/test_benchmarks.py?at=master&fileviewer=file-view-default#test_benchmarks.py-86

Here, both Storage and numeric_types are parametrized; currently, the test names look like this: test_getattr[instance-int32] where "instance-int32" is the param. What it would be REALLY useful is to:

1. save the infos about all params in the json, so I can later filter/aggregate based on those
2. be able to group by a specific param; e.g., --benchmark-group-by=param:numeric_type (or, even better: --benchmark-group-by=numeric_type)

I started to implement the feature here: https://github.com/antocuni/pytest-benchmark/commit/339984957de9f27db0120121abf4a9934671c777

but before going on, I'd like to see what you think. In particular, what do we do with the current "param" attribute? Do we keep it side by side with "params", or remove it completely?

[ionelmc]

I like the idea of being backwards compatible - we should keep the old --benchmark-group-by=param, for most cases (I'd expect people would have a single parametrization) it's quite fine.

I think allowing --benchmark-group-by=<something-from-tests> could allow collisions/confusion, --benchmark-group-by=param:<something-from-tests> looks more clear and easy to understand.

If we're to shorten things up then adding shorthand forms like --bm-group-by is a better alternative.

[antocuni]

ok, done :). See my PR: https://github.com/ionelmc/pytest-benchmark/pull/37 I agree that --benchmark-* options are too long to type, and I'm +1 on shortening all of them to --bm-* or --bench-*