Unsolvable pypi dependencies if platform excluded by feature

[baszalmstra]

The following manifest supports several platforms but the test environment only supports linux-64:

[project]
name = "project"
channels = ["conda-forge"]
platforms = ["linux-64", "osx-arm64", "win-64", "osx-arm64"]

[dependencies]
python = "3.11.8"

[pypi-dependencies]
timeago = "==1.0.16"

[feature.test]
platforms = ["linux-64"]

[environments]
test = ["test"]

All environments also have pypi-dependencies (because they are in the default feature), this means that to be able to solve this project we need a Python interpreter at runtime. The problem is that the test environment only supports linux-64 which means that on any other platform, there is no Python interpreter available and you get this error (after #1052 is merged):

x Unable to solve pypi dependencies for the test environment because no compatible python interpreter can be installed for the current platform
   ,-[4:13]
 3 | channels = ["conda-forge"]
 4 | platforms = ["linux-64", "win-64"]
   :             ^^^^^^^^^^^|^^^^^^^^^^
   :                        `-- even though the projects does include support for 'win-64'
 5 |
   `----
    ,-[13:13]
 12 | [feature.test]
 13 | platforms = ["linux-64"]
    :             ^^^^^^|^^^^^
    :                   `-- feature 'test' does not support 'win-64'
 14 |
    `----

I don't know what the solution to this problem would be. Perhaps we create a subset "solve environment" that just includes python which we can try to install to solve pypi-dependnencies regardless of the platform? That would also alleviate some of the issues reported in https://github.com/prefix-dev/pixi/issues/1046.

[anjos]

I'm hitting this problem simply because I'm using a pyproject.toml manifest which contains a project.dependencies entry.

I understand pixi assumes PyPI-dependencies exist in this condition, migrated automatically from the project's dependencies entry, even in the case they are also declared on the default feature dependencies entry. I fully understand this is a limitation, due to naming differences between Python and conda packages.

Now, this project happens to depend on pytorch. So, I followed the multi-env procedure to try to setup a CUDA-enabled environment for production as described in the pixi manual.

Unfortunately, that won't work due to this issue.

I guess this problem would hit more often users of pyproject.toml-based pixi manifests due to the automagic pypi-dependencies injection.

May be the current error message could be updated in this regard by looking at the manifest type and suggesting a more appropriate action (e.g. do not use a pyproject.toml in this case).

[anjos]

One question regarding this one: what would be the best way to mitigate this issue in your opinion? Having a separate directory with a pixi.toml and pixi.lock with the dedicated environment?

[tdejager]

As I a said in a previous issue I think there are two approaches. Let me quote myself:

1. Be able to define that you only want wheels in a environment, not super useful currently as an editable is not a wheel, so you need the interpreter for building. But potentially useful in CI if the wheel is built seperately or in a sperate env at least. That way you could record it as a direct-url dependency.
2. Have an sdist build environment specification in pixi. This would be a shared environment that is just used for building wheels. We only need to resolve and install this once, or not a lot of times at least.

For 2) I think it makes sense to specify this seperately :)

[tdejager]

I'm hitting this problem simply because I'm using a pyproject.toml manifest which contains a project.dependencies entry.

I understand pixi assumes PyPI-dependencies exist in this condition, migrated automatically from the project's dependencies entry, even in the case they are also declared on the default feature dependencies entry. I fully understand this is a limitation, due to naming differences between Python and conda packages.

Now, this project happens to depend on pytorch. So, I followed the multi-env procedure to try to setup a CUDA-enabled environment for production as described in the pixi manual.

Unfortunately, that won't work due to this issue.

I guess this problem would hit more often users of pyproject.toml-based pixi manifests due to the automagic pypi-dependencies injection.

May be the current error message could be updated in this regard by looking at the manifest type and suggesting a more appropriate action (e.g. do not use a pyproject.toml in this case).

Note that the dependencies that are in the pyproject.toml are parsed as pypi-dependencies, so it's not really assumed that there are pypi-dependencies. These are actually parsed as pypi-dependencies. You can use the custom pixi.tool.dependencies section to specify conda ones.

[anjos]

I'd like to document a workaround so that one can have a single manifest file, with CUDA support only for linux-64 while the project has pypi-dependencies and supports more than linux-64 as a platform (e.g. osx-arm64):

[tool.pixi.feature.cuda]
channels = ["nvidia", {channel = "pytorch", priority = -1}]
platforms = ["linux-64", "osx-arm64"]

[tool.pixi.feature.cuda.system-requirements]
cuda = "12.1"

# here is the trick - feature.cuda.dependencies is kept empty
# but we inject linux-64 dependencies with the target table
[tool.pixi.feature.cuda.target.linux-64.dependencies]
cuda = { version = "*", channel = "nvidia" }
pytorch-cuda = { version = "12.1.*", channel = "pytorch" }

[tool.pixi.environments]
cuda = { features = ["cuda"] }

The trick is to make sure the feature has no dependencies, and that the dependency table for linux-64 is injected via the target table for said feature. This was heavily inspired by the pixi documentation on the feature table (here: https://pixi.sh/latest/reference/configuration/#the-feature-table), with the sole exception of that detail. As is, the documentation does not work for due to (what I think is) this issue.

[ruben-arts]

He @anjos, the documentation contains a pseudo example, but feel free to improve it where you see fit. I'm not sure the change you propose is related to this issue.