tox needs a configuration schema validation

[ssbarnea]

Having a configuration file schema would clearly make tox easier to use.

• Users would know right away if they had a typo, like the famous skipdist vs skipsdist.
• With a published schema, code editors can verify it on the fly and implement auto-complete

While this ticked may remind us about https://github.com/tox-dev/tox/issues/999 I do not think that changing configuration format should be required in order to implement schema validation.

Making a new configuration format would be out of the scope but I am wondering if we could learn from http://schemastore.org/json/ which covers both json and yaml. One such validator for vscode can be seen at https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml -- I used it to validate some other file formats.

Maybe we could extend the existing validators to also use these schema format for validating ini files.

[gaborbernat]

I'm on board with this. I would probably handle this post the rewrite I'm doing now, that also separates schema files.

[jtrakk]

Marshmallow 3.X does a good job. It lets you define a schema, then validate and load nested dicts into objects (it's independent of serialization format, so it works with ini, toml, yaml, json, ...). It also has exporters for other specification tools like jsonschema.

[ssbarnea]

I am biased towards producing something that can be used by already existing code-editor plugins as I do not want to see ourselves hoping that people will start to write schema validators for tox. Out goal should be to publish a schema in a format that is easy consume by others. This can be quite tricky for tox because some options are implemented by plugins. This means that producing a relatively full schema would require to install all trusted plugins (we can keep an in-house list, probably).

[gaborbernat]

@ssbarnea feel free to do something now if you have the availability and time for it 👍 I'll definitely not have time to look at this more in detail this year, maybe next year myself, but if you do something I can do a review on it 👍

[ssbarnea]

@gaborbernat What is the status with the rewrite? I am asking this because soon this bug will be one year old and I still find lots of errors related to skipdist or similar typos.

I did not do anything about it for two main reasons:

• avoid conflict with your rewrite
• no indication about which library should be used to implement validation

[gaborbernat]

In progress, but slowly due my availability being tied down with virtualenv maintainer tasks... The rewrite is still aimed for the first version in September...

[jayvdb]

tox.ini has no valid syntax ;-) (in v3 at least)

See https://github.com/tox-dev/tox/issues/1702 and https://github.com/tox-dev/tox/issues/1708 for lots of unspecified and unexpected behaviour. They existing both at the top level ini structure, such as section names, and also finer details like trying to encode literals like \{.

The most prominent and easiest problem is that there are some restrictions on env names, however they are ignored when tox is called with -e or TOXENV, etc, etc. If _FACTOR_LINE_PATTERN, _ENVSTR_SPLIT_PATTERN and _ENVSTR_EXPAND_PATTERN were enforced uniformly, there would be quite a few edge use cases by real users that would be broken.

Even sane restriction on valid characters in paths would also impact envname which is used in paths.

[jayvdb]

I see https://github.com/tox-dev/tox-ini-fmt hasnt been mentioned here yet. I guess it is a good spot to develop validation of tox.ini. It should probably be added to the docs.