use of installed packages means poetry lock does not fully respect designated source for dependencies

[jclerman]

• Poetry version: 1.5.1
• Python version: 3.10.12
• OS version and name: MacOS 12.6.8
• pyproject.toml:

• [x] I am on the latest stable Poetry version, installed using a recommended method.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have consulted the FAQ and blog for any relevant entries or release notes.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below. N/A

Issue

When I started writing this, I thought the issue was different - now I think I see what's happening.

In a clean project for which poetry.lock has never been run before, poetry works as expected.

However, the results of poetry lock seem to be influenced by poetry's cache (somehow), since in an existing project, where poetry has successfully done lock before, it does not seem to be possible to tell poetry to limit where it looks for a given dependency.

Let's say we have the following pyproject.toml in a clean, new project, and that my_dep is only available in private repo privaterepo:

[tool.poetry]
name = "my-project"
version = "0.1.0"
description = ""
authors = ["Jeff Lerman <[email protected]>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10.0"
my-dep = {version = "^0.8.2", source = "PyPI"}

[[tool.poetry.source]]
name = "private_repo"
url = "https://repo.shared.privaterepo.com/repository/pypi-privaterepo/simple"
priority = "explicit"

[[tool.poetry.source]]
name = "PyPI"
priority = "supplemental"

[build-system]
requires = ["poetry-core>=1.4.0"]
build-backend = "poetry.core.masonry.api"

poetry lock correctly tells us:

Because my-project depends on my-dep (^0.8.2) which doesn't match any versions, version solving failed.

But now if we:

1. change the pyproject.toml to indicate the correct source (private_repo)
2. run poetry lock (completes successfully)
3. run poetry install (this is key!!)
4. change pyproject.toml back to the wrong source (PyPI)
5. run poetry lock

Then there is no failure anymore. We get the following clause in the resulting poetry.lock:

[[package]]
name = "my-dep"
version = "0.8.3"
description = "this is a very private package"
optional = false
python-versions = "*"
files = []

So, poetry is seeing the dependency in its local cache (somehow), using that to decide that it's a valid package, but not getting any metadata for it except the version & description - and populating an awfully sparse-looking poetry.lock clause.

That means there's no way to tell poetry, once a package has been found and installed (presumably, from any repo) that now we want to install it from a specific private repo - also no way to ensure that we've completely locked down the source for packages that we want to only ever come from our private repo.

I noticed this when trying to upgrade existing projects to the new poetry 1.5.x style of specifying sources, and not getting failures when I should have been.

[dimbleby]

run poetry install (this is key!!)

probably nothing to do with the cache then, which you can verify by clearing the cache before the next step

[dimbleby]

The responsible code is https://github.com/python-poetry/poetry/blob/eb74d6209ecda2840ea1c0c2375cb7f6485a39f9/src/poetry/puzzle/provider.py#L312-L313

This looks intentional, the comment for search_for_installed_packages() says

       Search for installed packages, when available, that satisfy the given
        dependency.

        This is useful when dealing with packages that are under development, not
        published on package sources and/or only available via system installations.

and I see some sense in that, though I also see a case for not doing this.

Seems like enough of an edge case that I personally don't care about it! but if any reader does then this should give them a start.

[jclerman]

I see - thanks for the pointer!

I confirm that poetry lock properly complains again once I do pip uninstall my-dep.

I think there are two issues here:

1. The poetry docs (https://github.com/python-poetry/poetry/blob/eb74d6209ecda2840ea1c0c2375cb7f6485a39f9/docs/repositories.md?plain=1#L128-L133) don't acknowledge that currently-installed packages are included in the search; they should be updated.
2. It seems like there should be a way to tell poetry to skip using locally installed packages, so that it is truly respecting the package-sources specified (so clearly, in poetry 1.5.x!) in pyproject.toml. Using locally installed packages without even a warning is... surprising.

It's especially troubling that poetry allows a way to specify a source for a dependency, but then simply ignores that source if a package by the same name (didn't necessarily come from the same source!) has been installed locally.

[dimbleby]

if you care about any of this then - by a huge margin - the most likely way to effect change is to submit a merge request.

docs updates in particular should be accessible to all.

[shubb30]

I believe I am also running into this issue. I have some custom packages in a private repo, and they were working before upgrading to Poetry 1.5 (using the secondary = true setting in the source).

Now that I've changed the source to priority = "explicit" my custom packages are not found when running poetry update | lock.

I confirmed that changing the source back to secondary = true allows it to find the package in my private repo.

[dimbleby]

not even close to being the same thing, please don't hijack this issue.

[domenix]

For our use case easy switching between private and public repositories for local package development is very important. I'd expect that if I delete the PyPi source from a package definition the next time I call poetry install it should pull it from the private repo instead of reusing the locally installed package (essentially invalidating it), so this is a blocking issue currently for adopting Poetry.

@jclerman Did you maybe have the chance to look into fixing this?

[jclerman]

Hi, no, I haven't had a chance to look into a fix unfortunately. The documentation update at least would be easy, and should probably be done first, but is if course not what we are really after here.