Add a dependency as editable and non-editable

[PeterBosch2]

This is about installing a dependency in --editable mode for development while using the same dependency in the CI/CD pipeline from the package repository.

I think this discussion was done several times before. However, I still do not see a good solution, unless I missed something badly. Even the recent change with a more relaxed install process (https://github.com/python-poetry/poetry-core/pull/520) does not do the trick.

To give some context: I am developing a data service in a data engineering project. My setup is to develop a service (proj-main) and some support libraries (proj-lib) with core data models and protobuf definitions. The development is carried out in two different workspaces. The file layout looks like this:

/workspaces/
  |- proj-lib/
  |- proj-main/

The split is due to the different scope and different artifact types: proj-lib is for downstream data users and shipped as a wheel. proj-main is to create a Docker image for deployment in k8s. proj-mainalso depends on proj-lib.

A pretty straight forward setup, I would say.

Having proj-lib added in editable mode (also in the git repository) leads to a broken pipeline, as proj-lib is not available in the next-door workspace in the build environment -- as we all know.

Adding proj-lib twice to pyproject.toml, one time as main dependency and the other time as dev dependence in --editable mode also does not do the trick, and -- absolutely correct -- leads to an error when configuring the setup due to version conflicts:

poetry add proj-lib
poetry add --group dev ../proj-lib --editable 

leads to

$ poetry add --group prod proj-lib
Using version ^0.7+g31a42f3 for proj-lib

Updating dependencies
Resolving dependencies... (0.0s)

Because proj-main depends on both proj-lib (0.0.0) @ file:///workspaces/proj-lib and proj-lib (^0.7+g31a42f3), version solving failed.

The relaxed install model for path dependency -- which I understood was meant to support this project layout -- does not help, as I need to add the dependency with the same name twice with different versions, which is obviously an inconsistent project description.

So my question: How to set this up in Poetry?

• From some of the discussions that I read, I got the impression that there is an opinion that this setup is invalid/uncommon, anyhow. -> What would be a better setup taking the scope consideration into account?
• Do I need to modify my pyproject.toml each time I check-in my proj-main?
• Is there a better way to realize this setup, e.g., with markers?

In case that I am not badly wrong in my assumptions that this is currently an impossible setup: Wouldn't that be a feature for poetry, e.g., to have a marker that I can set when calling poetry install to select the editable proj-lib dependency?

[PeterBosch2]

Any thoughts on this issue? Am I the only one to have this challenge?

[dimbleby]

you need to arrange that [poetry can tell that] your local copy of proj-lib has a version that's compatible with your main constraint. 0.0.0 is not such a version.

[ar4hc]

you need to arrange that [poetry can tell that] your local copy of proj-lib has a version that's compatible with your main constraint. 0.0.0 is not such a version.

Does not help really. (i'm on mac)

Error is:

database-connector @ file:///Users/.../src/.../database-connector (0.2.4)
database-connector (>=0.2.4,<0.3.0) ; source=mine

in pyproject.toml

database-connector = { path = "../database-connector", develop = true }
database-connector = { version = "^0.2.4", source = "mine" }

:(

[dimbleby]

@ar4hc source = "mine" is not compatible with a local path: only one of those things can be true