Change grouping defaults

[ionelmc]

By default all the benchmarks are grouped together (--benchmark-group-by=group, and the default for group is None).

I have opened this to open discussion about this as several people have complained about the grouping.

When I implemented the grouping this is what I had in mind:

• there should be grouping by default - in a way this advertises the feature and people won't ever wonder if pytest-benchmark has any grouping feature
• it's easy to change it by using --benchmark-group-by - similar to how most projects have some options in pytest.ini's addopts

Please add your thoughts about a better default and how it can outweigh the two considerations above.

[fjarri]

The kind of grouping I'd personally expect by default is grouping by testcase (I'm guessing that would be fullname?). Comparing the performance of different testcases seems like a niche thing. But having said that, I agree that I can just set that as a default in my project settings.

[max-sixty]

It would also be great to be able to have multiple dimensions in the grouping — for example, I test a few functions across array sizes. I'd like to benchmark by (function, array_size). Currently it looks like I can only benchmark by one of these, unless I create a weird fixture which unifies them and returns them as a tuple...