Thoughts on command-line vs configuration options

[nicoddemus]

Hi folks,

Currently in https://github.com/pytest-dev/pytest-xdist/issues/789 it has come up that users were asking to be able to configure --dist in the configuration file. While the usual solution is to use addopts for that, in that particular case it was not possible because pytest-xdist errors out if -n is not also passed, which is something you do not necessarily want in your configuration.

That is just an introduction, I'm writing this to put my thoughts on command-line vs configuration options and see what others think (I was writing this as an answer to Ronny in https://github.com/pytest-dev/pytest-xdist/issues/789 but decided to post this here to gather others' options).

Often in pytest we have options which are duplicated, where you have a command-line option which also has a dedicated configuration option for it.

As an example, we have the logging plugin: it has most of its options available in the command-line and also duplicated in the configuration file, which I think creates bloat (cognitive, documentation, code, etc). In many places in that plugin we have to deal with both flavors, and things can go wrong (forgetting to handle both somewhere, changing precedences, etc). IMHO I don't think the logging plugin should be considered an example/guideline on how options should be designed. We need to recall that the logging plugin was created a very long time ago, before pytest even had the -o option, so we inherited all the duplicated configuration.

Currently, users can easily hardcode command-line arguments via addopts, and override any configuration via the -o command-line argument, which I think it is great, and we should leverage that.

In my opinion, we should move our guidlines for options so that an option can only have one of two flavors:

• command-line for options which users frequently change between invocations.
• configuration file for options which are rarely changed, or are really part of the configuration of the test suite and will not change.

I definitely agree that addopts is not as elegant as a dedicated option, but I think it is the worth the tradeoff to avoid bloat.

Having a clear guideline on that would allow us to advertise more about -o (which to this day I'm surprised many people don't know about).

I do not have a strong opinion on this, just wanted to put my thoughts on writing to see if others feel the same, and if is worth having a guideline of sorts on this.

[RonnyPfannschmidt]

I created some initial prs for better ways to declare co-related options/cli args

Unfortunately it's often necessary to have both to provide a sane cli and a sane config experience

[RonnyPfannschmidt]

However I do believe that we also need better help output to accompany those options /cli args

[xmo-odoo]

For what it's worth, I think one of the issues is the ability to configure defaults for a plugin which is not required. xdist is a notable example of that (and the starting point of the issue), I might have a test suite for which xdist is not required but if used it has to be used a specific distribution, or at least default to that because otherwise the test suite is known to fail.

• With addopts that never works, you get an "unrecognized arguments" error and the test suite immediately fails.
• With ini you get an "Unknown config option" which is not the best but the test suite does run.
• If addini supported toml-style nesting as pseudo-namespacing (https://github.com/pytest-dev/pytest-xdist/issues/789#issuecomment-1324220843) it's currently forward compatible, pytest completely ignores the additional content.

And a lot of pytest plugins, especially the more generic ones published on pypi, are of that variety, they can be hugely helpful to improve runtime (xdist), tease out dependencies (randomly) or guard against flaky tests (rerunfailures) but they're often not necessary for the test suite to work.

Obviously this is a case by case basis, for some plugins the ability to configure defaults doesn't make much sense or is hardly load-bearing. But I think for xdist's dist it would be very helpful indeed.