Support Recursive Optional Dependencies

[tokoko]

Problem description

We're migrating to pyproject.toml and plan to switch to using recursive optional dependencies similar to what's described here. Seems like uv handles such dependencies correctly (during during compile and sync), but pixi update basically ignores additional extras when constructing a lock file. An example here.

[tdejager]

cc @baszalmstra as you've been the pixi update frontrunner :)

[ruben-arts]

@tokoko Are you able to create a minimal reproducer?

[tokoko]

@ruben-arts sure, I tested with the following just now:

[project]
name = "example"

[project.optional-dependencies]
np = [
  "numpy",
]

all = [
    "example[np]"
]

[tool.pixi.project]
channels = ["conda-forge"]
platforms = ["linux-64"]

[tool.pixi.environments]
np = {features = ["np"]}
all = {features = ["all"]}

I'm able to run pixi run -e np python -c "import numpy" just fine, but when I try pixi run -e all python -c "import numpy" I'm getting No module named 'numpy' error.

Here's the output from uv compile command (uv pip compile --extra all pyproject.toml):

# This file was autogenerated by uv via the following command:
#    uv pip compile --extra all pyproject.toml
numpy==2.0.2
    # via example (pyproject.toml)

Hope this helps

[ruben-arts]

Thank you, I would like to ask @tdejager for his input as this is a use-case I'm not sure we have tried to support yet.

[olivier-lacroix]

Thanks @tokoko for the example.

@ruben-arts @tdejager pixi init handles that case by resolving the recursion and writing

[tool.pixi.environments]
np = {features = ["np"]}
all = {features = ["all","np"]}

I am not sure this is fixable / should be fixed. This comes from the recursive nature of pyproject extras, versus the distinction in pixi between features and environments.

A middle ground may be to issue a warning during the parsing of the pyproject.toml in that case, hinting at adding the feature to the environment manually

[ruben-arts]

@olivier-lacroix Thanks for jumping in! Could we support it by just simply making it understand "self" and in the example case result in skipping example[np] in the dependency list?

[olivier-lacroix]

It is indeed what happens currently. example[np] is skipped.

And the np dependencies will only be considered if the feature is added to an environment.

[dhirschfeld]

I want to start using pixi for existing repos, using the pyproject.toml support but I'm running into this problem also.

All of our pyproject.toml files make heavy use of recursive optional dependencies to keep our deps DRY e.g.

dev = [
  "myproject[docs]",
  "myproject[lint]",
  "myproject[test]",
  "other-dep>=1.0",
]

This will unfortunately be a blocker for adoption. I can fix up the environments in my own repos but your average analyst here wouldn't have the necessary skills/knowledge (yet) 😞.

[tdejager]

@dhirschfeld ah that's unfortunate :(. Would dependency groups make sense for your use case? https://packaging.python.org/en/latest/specifications/dependency-groups/

[dhirschfeld]

Would dependency groups make sense for your use case?

I'm not sure - I've never used them! :thinking:

I'm not too sure of the motivation for dependency groups as it seems to just duplicate the recursive optional dependency functionality?

...but it seems more limited in that it can only be a collection of existing groups?

As shown in the example above, my extras are combinations of existing extras plus other packages.

Also, I'd like the extras to be installable - that doesn't seem possible with dependency groups?

There is no syntax or specification-defined interface for installing or referring to Dependency Groups. Tools are expected to provide dedicated interfaces for this purpose.

Finally, I will need my projects to be usable by both pip and pixi. The magic of pixi's pyproject.toml integration makes that possible - pip install -e .[test] works equally well as pixi shell -e test with no need to separately maintain dependencies for each tool... except for this issue.

[olivier-lacroix]

@dhirschfeld handling recursive extras would be possible indeed. In the meantime, you can use pixi init / replicate its behaviour by adding the various needed features to each environment. that is, in your case:

dev = ["dev", "docs", "lint", "test"]

[dhirschfeld]

That's what I'm doing for now.

It's actually working fine, as 90% of repos don't define their own custom extras groups and the owners of repos which do define custom extras groups are usually sophisticated enough to fix up their default pixi config.