pixi icon indicating copy to clipboard operation
pixi copied to clipboard

Support `--no-deps` option for pypi dependencies

Open lucifer1004 opened this issue 1 year ago • 17 comments

Problem description

--no-deps is not currently available. Hope it can be added.

lucifer1004 avatar May 20 '24 07:05 lucifer1004

Do you have an example use case in a pixi environment?

ruben-arts avatar May 21 '24 07:05 ruben-arts

For example, I want to install PyTorch via pip yet using locally installed versions of cuDNN and so on.

lucifer1004 avatar May 28 '24 06:05 lucifer1004

If you mean with local that you want to install it with conda dependencies we are aware that our mapping is not perfect yet as it can only map real python packages to conda packages and not "non-python" wheels like cudnn. But I don't see a reason for pixi to support depending on packages local to your machine yet.

ruben-arts avatar May 28 '24 06:05 ruben-arts

@ruben-arts I think you are misunderstanding. If someone is developing on cudnn then they don't want that dependency to be overridden by the pypi installation.

This can be achieved with --no-deps so that you only update the pytorch package, but don't modify the (existing) cudnn package. This is unrelated to the conda <-> pypi mapping.

Mamba / Conda / Pip also have this functionality.

The way I imagine this to work would be something like

foo = { version = "0.3.0", no-deps = true }

wolfv avatar May 28 '24 06:05 wolfv

Aah yes that makes more sense, thanks for the clarification @wolfv

ruben-arts avatar May 28 '24 07:05 ruben-arts

@wolfv Thank you for the clarification.

foo = { version = "0.3.0", no-deps = true }

is exactly what I am asking for.

lucifer1004 avatar May 28 '24 07:05 lucifer1004

I have another use case for this feature. Some Python packages have illy defined dependencies, for example PyTorch's tnt: https://github.com/pytorch/tnt/blob/5dad8d3638f9c967c9e9a351700481be5afa9b78/requirements.txt#L2

This pinned numpy's version excessively down to an obsolete version, which will clearly break the dep resolution. It would be great if pixi can provide an escape hatch to opt-in manual dependency control for a single package.

Yixuan-Wang avatar Jul 02 '24 20:07 Yixuan-Wang

+1 on this request, there are often poorly defined package dependencies causing dependency resolution to fail. To get around this i've been installing pip into the pixi environment and managing those packages via pip.

ihasdapie avatar Sep 30 '24 21:09 ihasdapie

I'd be keen to have this too! Might be a bit of a niche use case, but in projects that don't use pixi (yet) I like to have a personal (local-only) pixi setup. If the pyproject.toml file lists dependencies I want to prevent pixi from trying to reach pypi during install of mypkg = { path = ".", editable = true } because my environment doesn't have access to the internet.

E.g. there is currently no way to prevent the pypi fetch of pandas here afaik:

# pyproject.toml
[project]
name = 'mypkg'
version = '0.1.0'
dependencies = ["pandas"]


# pixi stuff
[tool.pixi.project]
name = "pixi-fr"
channels = ["conda-forge"]
platforms = ["linux-64"]

[tool.pixi.pypi-dependencies]
mypkg = { path = ".", editable = true }

Doing

- mypkg = { path = ".", editable = true }
+ mypkg = { path = ".", editable = true, no-deps = true}

would perfectly do the trick imo.

moritzwilksch avatar Oct 01 '24 13:10 moritzwilksch

+1 for the feature for when trying to get a sort of "workspace". Here, the editable installed should only rely on the dependencies of the default environment but pip will overwrite those.

[dependencies]
python = ">=3.10.0,<4"
flit = ">=3.9.0,<4"
pip = ">=24.2,<25"
cython = ">=3.0.10,<4"
compilers = ">=1.7.0,<2"
ninja = ">=1.12.1,<2"
meson = ">=1.5.1,<2"
meson-python = ">=0.16.0,<0.17"
numpy = ">=2.1.2,<3"
scipy = ">=1.14.1,<2"

[feature.dev.pypi-options]
no-build-isolation = ["joblib", "threadpoolctl", "scikit-learn", "imbalanced-learn"]

[feature.dev.pypi-dependencies]
joblib = { path = "./src/joblib", editable = true }
threadpoolctl = { path = "./src/threadpoolctl", editable = true }
scikit-learn = { path = "./src/scikit-learn", editable = true }
imbalanced-learn = { path = "./src/imbalanced-learn", editable = true }

glemaitre avatar Oct 15 '24 15:10 glemaitre

@glemaitre My workaround looks like this:

[feature.dev.tasks]
pixiinstall = "pixi install"
joblib = "pip install -e ./src/joblib --no-deps"
threadpoolctl = "pip install -e ./src/threadpoolctl --no-deps"
scikitlearn = "pip install -e ./src/scikit-learn --no-deps"
imbalancedlearn = "pip install -e ./src/imbalanced-learn --no-deps"

install = { depends-on = ["pixiinstall", "joblib", "threadpoolctl", "scikitlearn", "imbalancedlearn"] }

[environments]
dev = ["dev"]

Now a:

pixi run --environment dev install

should work.

pya avatar Oct 16 '24 07:10 pya

The other place this crops up in is in making use of various typing stub libraries. Some of these erroneously declare a runtime dependency on the library they are stubbing.

Allowing --no-deps would allow using these without the task workaround

mariusvniekerk avatar Oct 23 '24 14:10 mariusvniekerk

Another use case:

At my company we cannot release when there are known CVEs of high severity or higher, which I am 100% behind. However, sometimes a 3rd party dependency has a yet unfixed CVE, but we know that is is not used/imported in our current workflows. We would then wish to not install that unused dependency.

I know this is pretty niche, but it's yet another use case.

Note that currently in the uv codebase you can only specify --no-deps. They call it DependencyMode::Direct for an entire solve AFAICS at least. To Implement this nicely, as has been requested in this thread, we would need to be able to specify this per dependency. If this was the case I would have already done this :) Just know this is the main reason for the delay.

Briefly looking into the solver seems that this should be do-able, not sure if there is a reason why its not implemented.

On another note, we want to support the editable case with pixi build and the workspace feature as well, so that hopefully also replace some use of this :)

tdejager avatar Oct 24 '24 07:10 tdejager

I'd be keen to have this too! Might be a bit of a niche use case, but in projects that don't use pixi (yet) I like to have a personal (local-only) pixi setup. If the pyproject.toml file lists dependencies I want to prevent pixi from trying to reach pypi during install of mypkg = { path = ".", editable = true } because my environment doesn't have access to the internet.

+1 for that

noamgot avatar Mar 05 '25 09:03 noamgot

+1 Would be great to have this!

jakublala avatar Apr 08 '25 18:04 jakublala

Maybe the best thing is to try and make a PR to uv to add this capability to the solver, this would alleviate the solve step. We could however decide to not install it's dependencies because we have full control over this. However, I don't know if this is going to be enough to satifisy the issues.

tdejager avatar Apr 10 '25 08:04 tdejager

I'd be keen to have this too! Might be a bit of a niche use case, but in projects that don't use pixi (yet) I like to have a personal (local-only) pixi setup. If the pyproject.toml file lists dependencies I want to prevent pixi from trying to reach pypi during install of mypkg = { path = ".", editable = true } because my environment doesn't have access to the internet.

E.g. there is currently no way to prevent the pypi fetch of pandas here afaik:

pyproject.toml

[project] name = 'mypkg' version = '0.1.0' dependencies = ["pandas"]

pixi stuff

[tool.pixi.project] name = "pixi-fr" channels = ["conda-forge"] platforms = ["linux-64"]

[tool.pixi.pypi-dependencies] mypkg = { path = ".", editable = true } Doing

  • mypkg = { path = ".", editable = true }
  • mypkg = { path = ".", editable = true, no-deps = true} would perfectly do the trick imo.

Any update to this request? I need the same thing with having a pypi-dependency from my local project, but the pixi has all the necessary requirements coming from conda channels already.

My workaround is going to be an activation script.

jtroe avatar May 19 '25 21:05 jtroe

+1 for this. It happens so often that people specify upper version bounds for no specific reason. Having such a poor defined dependency either keeps your whole env in a ridiculously outdated state or makes it impossible to solve at all. Therefore it might even make sense to have an:

{ deps-ignore-upper-bounds = true }

I realize this opens a potential can of worms, so I would be perfectly happy with:

{ no-deps = true } 

and adding the necessary (but more loosely pinned) dependencies of the dependency "by hand".

jorenretel avatar Jun 03 '25 12:06 jorenretel

@jorenretel I'd guess that for that use-case it would make more sense to override an individual dependency a la a: https://docs.astral.sh/uv/concepts/resolution/#dependency-overrides

tdejager avatar Jun 03 '25 12:06 tdejager

@jorenretel I'd guess that for that use-case it would make more sense to override an individual dependency a la a: https://docs.astral.sh/uv/concepts/resolution/#dependency-overrides

hi @tdejager, thanks, that seems exactly what I need!

But for the moment adding things to [tool.uv.override-dependencies] in my pyproject.toml (as the uv documentation you linked to suggests) does not seem to be used by pixi, according to the note at the bottom of this page: https://pixi.sh/latest/python/pyproject_toml/#example_1 . Do you have a suggestion to work around this current limitation?

Edit: okay, saw the other issue you made.

jorenretel avatar Jun 03 '25 13:06 jorenretel

Yeah sorry not really a workaround for now alas. I indeed made the other issue instead, to track this, don't have time to get to it soon but I can support is you'd like to try your hand at a PR :)

tdejager avatar Jun 03 '25 14:06 tdejager