Support matrix-based environment specifications

[mdekstrand]

Problem description

It would be useful (to me, at least) for Pixi to support matrix-based environment specifications, perhaps similar to Tox, for settings where we need multiple combinations of environments (e.g. multiple Python versions, each of which has multiple variants).

For simplicity (and removing the need to support variable interpolation in configuration files), it could be restricted so that each of the matrix specifier elements is itself a valid feature (or the empty string, to indicate no extra feature), and the environments produced by the matrix contain those features + a set of common features.

Maybe something like this:

[environment-matrices.test]
common-features = ["test"]

[environment-matrices.test.members]
python = ["py311", "py312"]
variant = ["", "cuda"]

expands to:

[environments.test-py311]
features = ["py311", "test"]
[environments.test-py311-cuda]
features = ["py311", "cuda", "test"]
[environments.test-py312]
features = ["py312", "test"]
[environments.test-py312-cuda]
features = ["py312", "cuda", "test"]

[ruben-arts]

FYI: We initially didn't want to automatically have this behavior to avoid unmanageable situations.

I think that there is something to say for this feature, especially if the user defines it as such.

So I would be open to merging a good design, the core team is currently at capacity with other projects. For a feature like this I would like more input on multiple use-cases and implementation details. Also pining @pavelzw @0xbe7a @olivier-lacroix as they have been big contributors to the multi-env work.

Some notes on this topic:

• We might need to improve the inspection tooling (pixi list pixi info etc.) to help with managing this kind of features
• variants are a common thing in the conda ecosystem for building, we're currently thinking about that for pixi build so these features might collide/work together.
• These matixes are often dependency specific (python, pytorch, rust), so for readability we might want to think of dependency specific variants.

[mdekstrand]

Make sense! I don't have any bandwidth to try to push on implementing such a thing, but would be happy to provide design feedback or test prototypes.

[hameerabbasi]

Related: https://discord.com/channels/1082332781146800168/1308668662256762952/1308721310360141874

Copying the text here for those not on Discord:

So you'd define a set of environments in a matrix.

The interface I envision is something like:

# Each dependencies gains a "matrix" property, `None` by default.
[feature.env.matrix-dependencies]
python = ["3.10", "3.11", "3.12"]
numpy = ["2.*"]

# This one is a list
[[feature.env.matrix-include]]
numpy = "1.*"
python="3.10"

# Similarly exclude

Then, one would do the following:

pixi run -e env[some_specifier_here] <task>

To specify which variant to run the task in. One could also maybe do (this is the useful part)

pixi run --matrix -e env[*] <task>

To run a task in the whole matrix -- thereby testing all specified configs.

---

Another related thought -- we might want to solve all the environments, but not really install all of them by default. Having something in the TOML or cmdline to limit which envs to install might make sense.

[baszalmstra]

I like this idea. The reason we didnt use matrices before is that the number of environments can quickly explode like crazy which all require solves. I think this is still an issue. But perhaps we could leave this decision up to the user.

We could also do:

[feature.env.dependencies]
python = [
	{ version = "3.10" },
	{ version = "3.11" },
	{ version = "3.12" }
] 

I don't think we would need the extra matrix-dependencies option.

[partrita]

When installing Python libraries using PyTorch, I often encounter code written for older versions (CUDA 11 and Python 3.9). What I'm looking for is handling environments with different Python and CUDA versions using pixi. Below is the pyproject.toml file I've created. There's a lot of repetitive content - is there a more elegant way to manage this?

[project]
authors = [{name = "NAME", email = "EMAIL"}]
dependencies = []
description = "Add a short description here"
name = "FOLDER_NAME"
requires-python = ">= 3.9"
version = "0.1.0"

[tool.pixi.project]
channels = ["conda-forge"]
platforms = ["linux-64"]

[tool.pixi.environments]
cuda11-py39 = ["cuda11-py39"]
cuda11-py312 = ["cuda11-py312"]
cuda12-py39 = ["cuda12-py39"]
cuda12-py312 = ["cuda12-py312"]
default = ["cuda12-py312"]

[tool.pixi.feature.cuda11-py39.system-requirements]
cuda = "11.0"

[tool.pixi.feature.cuda11-py39.dependencies]
python = "~=3.9.0"
cuda-version = "~=11.0"
pytorch-gpu = ">=2.5.1, <3"

[tool.pixi.feature.cuda12-py39.system-requirements]
cuda = "12.0"

[tool.pixi.feature.cuda12-py39.dependencies]
python = "~=3.9.0"
cuda-version = "~=12.0"
pytorch-gpu = ">=2.5.1, <3"

[tool.pixi.feature.cuda11-py312.system-requirements]
cuda = "11.0"

[tool.pixi.feature.cuda11-py312.dependencies]
python = "~=3.12.0"
cuda-version = "~=11.0"
pytorch-gpu = ">=2.5.1, <3"

[tool.pixi.feature.cuda12-py312.system-requirements]
cuda = "12.0"

[tool.pixi.feature.cuda12-py312.dependencies]
python = "~=3.12.0"
cuda-version = "~=12.0"
pytorch-gpu = ">=2.5.1, <3"

[hameerabbasi]

I like this idea. The reason we didnt use matrices before is that the number of environments can quickly explode like crazy which all require solves. I think this is still an issue. But perhaps we could leave this decision up to the user.

We could also do:

[feature.env.dependencies] python = [ { version = "3.10" }, { version = "3.11" }, { version = "3.12" } ]

I don't think we would need the extra matrix-dependencies option.

That sounds okay to me; I'd still add include/exclude in some form though.

[dhirschfeld]

For prior art on include/exclude there is the GitHub Actions matrix strategy: https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/run-job-variations