Non-editables and editables install cannot be combined in the same solve-group

[tdejager]

Checks

• [X] I have checked that this issue has not already been reported.

• [X] I have confirmed this bug exists on the latest version of pixi, using pixi --version.

Reproducible example

Given the following pyproject.toml:

[project]
name = "minimal_project"
version = "0.1"
description = "just says hello"
requires_python = ">=3.11"

[build-system]
requires = ["flit_core>=3.2,<4"]
build-backend = "flit_core.buildapi"

[tool.pixi.project]
name = "minimal_project"
channels = ["conda-forge"]
platforms = ["osx-arm64", "linux-64", "win-64"]

# Use the project in editable for development
[tool.pixi.feature.dev.pypi-dependencies]
minimal_project = { path = ".", editable = true}

# Don't install in editable mode for production
[tool.pixi.feature.prod.pypi-dependencies]
minimal_project = { path = ".", editable = false}

[tool.pixi.environments]
# We want to use the same solve-group for solving
# the same set of dependencies and we would only
# need to do this once
default = {features = ["dev"], solve-group = "default"}
prod = {features = ["prod"], solve-group = "default"}

Minimal project is just a simple example project, see the pixi/examples/pypi-source-deps for the module.

We have a case that @pavelzw has a lot, where we want to install a dependency as editable in development and non-editable in production. But do need to use the same solve-group so we use the same set of dependencies everywhere.

Currently running this will create both editables for development and production.

pixi list -e prod
Package          Version       Build               Size       Kind   Source
...
libzlib          1.2.13        h53f4e23_5          47 KiB     conda  libzlib-1.2.13-h53f4e23_5.conda
minimal_project  0.1                                          pypi   . (editable)
ncurses          6.4.20240210  h078ce10_0          801 KiB    conda  ncurses-6.4.20240210-h078ce10_0.conda
...

Removing the solve-group solves this.

pixi list -e prod
Package          Version       Build               Size       Kind   Source
...
libzlib          1.2.13        h53f4e23_5          47 KiB     conda  libzlib-1.2.13-h53f4e23_5.conda
minimal_project  0.1                                          pypi   . 
ncurses          6.4.20240210  h078ce10_0          801 KiB    conda  ncurses-6.4.20240210-h078ce10_0.conda
...

Issue description

So removing the solve-group solves the issue but is generally not what you want. What we want is to solve once and install in two different ways, this requires building two seperate wheels. Editable wheels are different from regular ones. But because we need to resolve an source essentially we need to build the metadata first in the resolution phase.

So there are two problems:

1. Using the same solve-group incorrectly marks the dependency as editable in the prod installation.
2. PEP517 defines a hook: prepare_metadata_for_build_wheel and PEP662 defines a hook prepare_metadata_for_build_editable, for which there are no guarantees that they produce the same metadata. Actually, we know they don't produce the same .dist-info folder for sure (missing RECORD amongst other things)

We just need to debug 1) to see whats going wrong. However, for 2) we need to assume that the hooks create the same METADATA file for now. It's the only way to allow this feature to work. I currently see no reason why it shouldn't generate the same METADATA but I don't know all the build backends internals or projects that might do something that produces different METADATA. So if anyone can enlighten me please!

Expected behavior

This should create two environments a default one with the editable install and a prod one where the installation is not editable.

[pavelzw]

or https://github.com/prefix-dev/pixi/pull/1121#issuecomment-2039816866

Technically you cannot install a package editable and non-editable at the same time; so it also kind-of makes sense to not allow this... (but it would make like in a docker deployment a bit easier)

[noamgot]

hi, any updates on this one? As far as I can tell it still can't be done, but I'd like to hear about recommended practices to overcome this issue.

[tdejager]

Hi @noamgot, unfortunately the best workaround is to manually install your package on top of solve group as is done in this example: https://github.com/prefix-dev/pixi/blob/main/examples/docker/pyproject.toml

We would need to think if there is some way we can solve it in pixi, its just that the .dist-info is a bit different between the editable and non-editable.

[noamgot]

@tdejager this is kind of what I came up with:

[tool.pixi.pypi-dependencies]
my-package = { path = ".", editable = true }
# ...
[tool.pixi.environments]
default = { features = [ "test", "dev"], solve-group = "default" }
prod = { features = ["prod"], solve-group = "default" }
# ...

# I need to pack my environment using conda-pack
[tool.pixi.feature.prod.tasks.pack-env]
# Before packing, we need to uninstall the current version of the package (which is installed as editable)
# and then install the package from the current directory.
cmd = """
pip uninstall my-package -y && \
pip install . && \
conda-pack -p $CONDA_PREFIX -o $CONDA_PACK_OUTPUT\
"""

But - looking at the example of the docker, I think it'd make more sense indeed to move the editable dependency to the dev feature.

---

I still have 2 follow-up questions:

1. In the CI we run tests in the test environment. I assume the existence of my source code is enough, but would it make sense to install the package (in non-editable mode)?
2. And for my conda-pack - what's the benefit of building the package (with hatch, for example) over installing with pip install .? In this case, at least, this is a development package I don't need to publish anywhere...

[pavelzw]

Btw https://github.com/Quantco/pixi-pack might also be interesting for you (also see the "injecting additional packages" section)

[noamgot]

@pavelzw I saw it, and it does a great job - but it doesn't support pypi packages (I have some 3rd party packages that are unfortunately only in pypi). Moreover - specifically for my use case, the packed environment is opened in a cloud container, and must be a conda-pack output (the docker image, as well as the unpacking process are not programmed by me; If it will be possible to use a custom docker image I'll probably use your informative blog post).

[cas--]

This is rather unfortunate bug, I spent quite a block of time trying to understand why the docker image with prod environment was missing the source files.

I wonder if pixi could at least warn about problems with both an editable and non-editable version of package until this is resolved?