Add option to skip installing path dependencies

[adriangb]

This serves the same purpose as --only-root but for path deps (which just like the "root" will bust caches if any source file changes).

If/once https://github.com/python-poetry/poetry-core/pull/520 gets merged this will allow writing a Dockerfile along the lines of:

FROM python

COPY pyproject.toml poetry.lock .
RUN pip install poetry && poetry install --no-root --no-path
COPY src/ ./src
COPY folder-of-path-deps/ folder-of-path-deps/
RUN poetry install

I chose --no-path because of it's parallels to --no-root, but I'm open to other options.

Closes #6845 Closes #6850 Closes #3671

[adriangb]

I think this fixes https://github.com/python-poetry/poetry/issues/668

[adriangb]

@radoering while we wait for review on https://github.com/python-poetry/poetry-core/pull/520, would you mind taking a look at this and giving some thoughts? Thanks!

[radoering]

If I understand correctly, the main motivation for this feature is the Docker cache. Before taking a look into the implementation details, I'd like to here the opinions of other maintainers if this feature makes sense for them. Especially @neersighted because he has much more experience with Docker than I do.

Just one question: Does it make sense to lump directory and file dependencies together (since both are path dependencies) or should it only consider directory dependencies?

[adriangb]

Just one question: Does it make sense to lump directory and file dependencies together (since both are path dependencies) or should it only consider directory dependencies?

Great question. I’m not super familiar with workflows around file dependencies, I’ve never used them. The fundamental question as far as caches are concerned is “does this likely change every build (source code) or only rarely (dependencies)?”. The main use case I associate file dependencies with is vendoring, in which case they’re not changing any more often than other leaf dependencies. What makes directory dependencies special is that they’re often actually your own source code. We could always add another flag allowing users to choose depending on their use case. It’s a bit more complexity but avoids painting ourselves into a corner if we miss some use cases.

[neersighted]

I'm not convinced this is useful, as in the use-case shown here (not building a package, just using Poetry to set up an environment), groups could easily be used. I don't think path dependencies should be special at all; encouraging users to think of them as 'special' takes us down a path we have strictly wanted to avoid (e.g. leaking metadata beyond the Core Metadata spec between layers of the dependency tree because Poetry is used all the way down), and reduces the interoperability of Poetry with the rest of the ecosystem.

I think that documenting the use of this pattern, but with groups, can help the people who are trying to do monorepos today, and that a workspace-like approach like suggested in #2270 is what we should explore, instead of trying to tear down the Core Metadata/PEP 517 wall between Poetry projects.

[adriangb]

in the use-case shown here (not building a package, just using Poetry to set up an environment), groups could easily be used

could you explain how? I use dependency groups in conjunction with path dependencies (each group is just a path dependency thus allowing selecting/deselecting subprojects; see https://github.com/adriangb/python-monorepo/tree/main/poetry) in a real world monorepo and it works great aside from docker caching. I don’t see how “using dependency groups” solves the same problem this PR does?

[neersighted]

Hmm, I did not realize that this PR was targeted at the pattern in your example repo there. While I brought up the overall direction for better supporting monorepos as kind of a extension of my thinking re: this PR, I did not realize that is at the core of what you want to do.

I think that makes my objection firmer -- special support for using path dependencies in this way (with a superproject that delegates to subprojects) is not desirable because it is ultimately incomplete unless we tear down the metadata barrier, and all the designs along that line have been rejected to date. It's possible we decide that the subproject approach is not tenable and go that way after all, but I think it would have to be comprehensive and not piecemeal. It would probably be better if it was a new dependency type if that were the case, instead of overloading path =, but I digress.

Regarding the use case you have, I'm afraid I still don't see how what you want to do is not possible with dependency groups. Could one not do the following (once https://github.com/python-poetry/poetry-core/pull/520 is in, with all your groups set to optional)?

FROM python:3.11

COPY pyproject.toml poetry.lock .
RUN pip install poetry && poetry install --no-root
COPY src/ ./src
RUN poetry install --only-root
COPY folder-of-path-deps/ folder-of-path-deps/
RUN poetry install --no-root --with <path-dep-groups>

[adriangb]

special support for using path dependencies in this way (with a superproject that delegates to subprojects) is not desirable because it is ultimately incomplete unless we tear down the metadata barrier, and all the designs along that line have been rejected to date

I haven’t heard any objection to my design. If you can think of any issues let's discuss in #6850.

I think it would have to be comprehensive and not piecemeal

Yes this is a small isolated piece, but that is by design. Part of the attractive of this design is that it can be implemented in small incremental steps, most of which except maybe this one are already completed and we’re beneficial to Poetry regardless of the acceptance of the proposed monorepo design. So I wouldn’t really call it “piecemeal”.

I'm afraid I still don't see how what you want to do is not possible with dependency groups. Could one not do the following (once https://github.com/python-poetry/poetry-core/pull/520 is in, with all your groups set to optional)?

I don’t think things will work properly without this PR but I’m away from a computer to confirm, I can test later or feel free to give it a crack with my example repo.

[adriangb]

Ok so I had a chance to look at your example and test in my example repo.

The issue with the solution you proposed is that you never install the transitive 3rd party dependencies (which may even be all 3rd party dependencies if there is no "root" pacakge/dependencies), so there's no layer that has "all of the things that are expensive to install but don't change often" which is what we want for good caching.

I updated my example repo to use this branch (https://github.com/adriangb/python-monorepo/commit/0250ddac28d8cad20381948efad6eae3061f99be) and tested that indeed doing poetry install --only app --no-path gives me transitive dependencies of app (in particular, app depends on lib and lib depends on pydantic and indeed pydantic gets installed).

So at least as proposed dependency groups are not an alternative to this PR.

[adriangb]

pre-commit.ci autofix

[adriangb]

@radoering I renamed path to directory as per your feedback in #6850

[adriangb]

I think this will also be useful in CI workflows (so not Docker but similar) since the same principle of transitive deps changing less than source code apply.

[taynaud]

Maybe this change is more about top level paths dependancy than directory or files (i.e from the project.toml rather than transitive path dependancies) ? In this case, the option could be --no-root-paths or --no-top-level-paths ?

[adriangb]

It does skip transitive dir dependencies

[mwgamble]

The issue with the solution you proposed is that you never install the transitive 3rd party dependencies (which may even be all 3rd party dependencies if there is no "root" pacakge/dependencies), so there's no layer that has "all of the things that are expensive to install but don't change often" which is what we want for good caching.

Can confirm, this behaviour is essential for making this work in a useful manner. I tried the package groups functionality when I was trying to make this work nicely, and it just didn't work properly at all because of the transitive dependencies issue.

[adriangb]

@rodering, I addressed all of the feedback and got the branch in sync with the default branch. Apologies if the commits got messy, I'm working from plane wifi.

[github-actions[bot]]

Deploy preview for website ready!

✅ Preview https://website-gj3tbks72-python-poetry.vercel.app

Built with commit f41c0350792493b797f11d658bbe395092d468c0. This pull request is being automatically deployed with vercel-action

[adriangb]

Thank you for pushing this across the finish line Randy!

[github-actions[bot]]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.