Allowing --benchmark-skip and --benchmark-only

[The-Compiler]

What do you think about allowing both --benchmark-skip and --benchmark-only and giving the latter precedence? That'd mean one could have --benchmark-skip in pytest.ini in order to never run benchmarks on normal testruns, and add --benchmark-only on the commandline for benchmark runs.

[ionelmc]

Seems dangerous to skip tests by default (people wouldn't run them often => they become useless).

A better way perhaps is to allow --benchmark-disable (in your pytest.ini, still runs the tests using benchmark) and --benchmark-only (or a new --benchmark-enable) to re-enable them again.

[The-Compiler]

Ah, I didn't consider tests which actually test stuff and do benchmarking. --benchmark-disable would then run those and not do any benchmarking, right? That seems what I want. I might submit a PR at some point, but as usual, there's lots more stuff to do than time :wink:

[ionelmc]

Note that --benchmark-disable is automatically activated if xdist is on or you're missing the statistics dependency. This means that a --benchmark-enable option should be the only way to "possibly enable benchmarks" (to leave --benchmark-only have an error path if benchmarks can't be run).

[Suor]

I think tests and benchmarks should be separated. Can't think of a situation when I want to run both tests and benchmarks in a single call. This is being the default behavior puzzles me.

[ionelmc]

Uh .. so I've checked the code, --benchmark-enable overrides --benchmark-disable since b181664c407b4c30354baaa89fde76057bdb71d1.

Can we just close this and leave it at that?