Does not cohabitate with other envlist filters well

[greg-hellings]

Having both tox-factor and tox-travis living together, for instance, causes one to override the other. Since they both read the full list of environments directly from the config, whichever one gets executed last will override the filtering of the other.

Should, instead, operate on the selected envlist

[ryanhiebert]

Oh that's interesting. Doing that seems like a reasonable idea. I'm not immediately sure how reasonable an implementation would be.

[greg-hellings]

It should be as straightforward as changing from reading the tox.ini to generate the list of all environments internally (although https://github.com/tox-dev/tox/blob/e639a54eab5641c8dde353a29a516e4bf5487e37/src/tox/session/commands/show_env.py#L7-L10 gives the much more straightforward method that tox uses to generate the full list internally) to just operating on config.envlist, no? Filter it down to only those that match the current system python value?

That avoids both running environments that are not specified in the default envlist that people might not want run explicitly and also allows it to cohabitate with other filter plugins that also respect the config.envlist.

[rpkilby]

Hi @greg-hellings. Could you give an example of how you intend to use tox-factor and tox-travis together? For context, tox-factor is intended to be used in lieu of tox-travis, and it is largely intentional that they are not compatible.

tox-travis is more implicit, selecting testenvs based on matching environment variables/factors. In contrast, tox-factor requires that factors are provided explicitly. To mix and match these behaviors would seem somewhat counterintuitive to me. e.g., if you wanted to use tox-travis to match the Python version, then tox-factor to subselect against some other factor, then the obvious solution to me would be to just provide the Python version as one of the factors (tox -f py37-other-factors).

That all said, what usage did you have in mind?

[greg-hellings]

My test cases have a matrix with at least 4 axes (a repository of Ansible roles, in particular). The axes are python version, Ansible version, role, scenario. In Travis CI the goal is to split a job along each of those 4 axes as each represented by an environment variable in the .travis.yml file.

I would like to generate a tox env for each axis, then have them filtered appropriately. Currently in tox-travis I can get automatic filtering of the python version axis, but I can't just specify a list of environment variables to filter further. Tox-factor gave me the ability to specify which additional environment variables to filter on, but not to automatically filter the Python version like this plug-in.

I know that I can specify matrix variables here with the syntax

[travis:env]
ROLE =
  role_a: role_a
  role_b: role_b

However, that introduces a nightmare of maintenance where I need to redefine the same list in .travis.yml and again here in tox.ini. It would be great to have the ability to just say something like

[travis]
factor_variables =
  ROLE
  SCENARIO
  ANSIBLE

and have those be filtered the same as the previous syntax but just with the values passed through. tox-factor gave me that, but since both that and this operate on the full list of all defined environments and because they parse it straight from the tox.ini file instead of the config.envconfigs.keys() list, it makes it impossible to get the two to live together.

[rpkilby]

I don't really follow what you're doing, but if tox-factor works for the other three axes, why can't you let tox-factor also handle the Python axis?