solve-group with conflicting requirements in different envs

[rgommers]

Checks

• [X] I have checked that this issue has not already been reported.

• [X] I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

I'm not sure if this is a bug report or a feature request. There are some issues/discussions that looked similar (e.g., https://github.com/prefix-dev/pixi/discussions/1817), but they usually had to do with mixing conda and pypi dependencies.

Use case: multiple envs with a common set of dependencies that must be the same in all envs, and different possibly conflicting extra dependencies in separate envs.

Minimal example, installing numpy plus a BLAS library, showing the solve error:

[project]
name = "check-solve-group"
version = "0.1.0"
description = "Minimal reproducer of solve-group with conflicting dependencies"
authors = ["Ralf Gommers <[email protected]>"]
channels = ["conda-forge"]
platforms = ["osx-arm64", "linux-64"]

[dependencies]
blas-devel = "*"
numpy = ">=2.0.0,<3"
threadpoolctl = ">=3.5.0,<4"

[tasks]
check-blaslib = "python -c 'import numpy as np; print(np.show_runtime())'"


[feature.openblas.dependencies]
libblas = { version = "*", build = "*openblas" }
openblas = ">=0.3.27,<0.4"

[feature.mkl.target.linux-64.dependencies]
libblas = { version = "*", build = "*mkl" }
mkl = ">=2023.2.0,<2025"

[feature.netlib.dependencies]
libblas = { version = "*", build = "*netlib" }


[environments]
default = { features = ["openblas"], solve-group = "bld" }
netlib = { features = ["netlib"], solve-group = "bld" }
mkl = { features = ["mkl"], solve-group = "bld" }

% pixi ls
  ⠐ bld:linux-64         [00:00:02] applying JLAP patches                                                                                                                   × failed to solve the conda requirements of 'bld' 'osx-arm64'
  ╰─▶ Cannot solve the request because of: The following packages are incompatible
      ├─ libblas * *openblas can be installed with any of the following options:
      │  └─ libblas 3.9.0
      ├─ openblas >=0.3.27,<0.4 can be installed with any of the following options:
      │  └─ openblas 0.3.27 | 0.3.27 | 0.3.27 | 0.3.28
      └─ libblas * *netlib cannot be installed because there are no viable options:
         ├─ libblas 3.9.0, which conflicts with the versions reported above.
         └─ libblas 3.9.0 | 3.9.0 | 3.9.0 | 3.9.0 | 3.9.0 | 3.9.0, which conflicts with the versions reported above.

If I leave out solve-group or use different group names, things work as expected:

% pixi ls | rg libblas
libblas          3.9.0     23_osxarm64_openblas  14.7 KiB   conda  libblas-3.9.0-23_osxarm64_openblas.conda

% pixi ls -e netlib | rg libblas
Environment: netlib
libblas          3.9.0     6_h4c1867c_netlib   164.9 KiB  conda  libblas-3.9.0-6_h4c1867c_netlib.conda

Issue description

I had expected solve-group to handle conflicting dependencies as long as those are not present in the same environment. There are quite a few use cases for this, typically comparing different alternatives/libraries (for performance, behavior, etc.). E.g.:

• Use and compare different BLAS libraries
• Switch from CPU to CUDA versions of one library (e.g., PyTorch), keeping everything else unchanged
• Comparing different array libraries (e.g., PyTorch vs. JAX) which may not easily be installable in the same env
• Comparing two versions of the same package, with everything else unchanged

For a longer description of those real-world use cases, see https://github.com/rgommers/pixi-dev-scipystack/tree/main/scipy.

This seems like a need that is common, for debugging, benchmarking, supporting/testing extra functionality, etc. The downsides of not having a guarantee that two envs have a common subset of packages include:

• When running the same script or dev task in two envs, it is much harder to ascribe behavior or performance differences that one observes to the feature(s) that differ between the two envs.
• If a build step is involved in the dev task, having different common dependencies may trigger unnecessary rebuilds or even outright fail if one uses the same build directory.

Expected behavior

Common dependencies between environments are guaranteed to be the same version in every environment they occur in when they're in the same solve-group. And the solve works for conflicting dependencies that are present in separate environments.

[tdejager]

@baszalmstra any smart remarks here :)?

[baszalmstra]

At the moment this is not how solve groups work. They are used to group requirements together into one large environment and then extracting sub-environments

Having a feature like you suggest would be very useful but is also quite complex. Perhaps this is something we can add later!

[rgommers]

Yeah that's what I figured, thanks for confirming @baszalmstra. I guess we're going to rely on the solver picking the same versions independently for now. No guarantees of that happening, but it often does happen as long as no new dependency in a non-default env placed an upper bound on a version in the default env.