PyPI (pypi.org) gets added as an index even though it's overwritten in PDM's pyproject.toml

[MikulasZelinka]

Describe the bug PyPI (pypi.org) gets added as an index even though it's overwritten in pyproject.toml using the pypi name in unearth>=0.13

I guess it could be caused by https://github.com/frostming/unearth/commit/7d1c215 (although I don't really see how it's possible looking at the source code)...

Version 0.12.1 works fine and does not have this issue.

I'm not sure if this is a pdm or an unearth issue.

To Reproduce

[[tool.pdm.source]]
name = "pypi"
url = "https://<some_custom_artifactory_url>/artifactory/api/pypi/pypi-all/simple"
verify_ssl = true
include_packages = ["*"]

pdm install -vv --prod --no-lock --check --no-editable --no-self --fail-fast

Now the log of install looks like:

18:06:22  #12 2.285 Run command: ['/usr/local/bin/python3.11', '-m', 'virtualenv', '/<package>/.venv', '-p', '/usr/local/bin/python', '--prompt=<package>-3.11', '--no-pip', '--no-setuptools', '--no-wheel']
18:06:22  #12 2.****78 created virtual environment CPython3.11.6.final.0-6**** in 89ms
18:06:22  #12 2.****78   creator CPython3Posix(dest=/<package>/.venv, clear=False, no_vcs_ignore=False, global=False)
18:06:22  #12 2.****78   activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
18:06:22  #12 2.****99 Virtualenv is created successfully at /<package>/.venv
18:06:32  #12 11.62 STATUS: Fetching hashes for resolved packages...
18:06:32  #12 11.67 Synchronizing working set with resolved packages: 100 to add, 0 to update, 0 to remove
18:06:32  #12 11.67 
18:06:32  #12 11.88 unearth.collector: Collecting links from https://pypi.org/simple/anyio/
... (repeats many times for different packages)
18:06:32  #12 11.89 unearth.auth: Found credentials in index url for pypi.org
18:06:33  #12 13.00 unearth.collector: Fetching HTML page https://pypi.org/simple/annotated-types/
18:06:33  #12 13.00 unearth.collector: Collecting links from https://<some_custom_artifactory>/artifactory/api/pypi/pypi-all/simple/annotated-types/
... (artifactory links are mixed in for the packages too, but in the end downloads from pypi are preferred)

Expected behavior The only index should be the some_custom_artifactory_url. But instead, both PyPI and this custom index are used.

It also looks like PyPI is preferred, and the custom artifactory is used iff a package is not found in PyPI.

System (please complete the following information):

• unearth version: >0.12.1
• Python version: 3.11
• OS: Linux
• pdm version: 2.11.1

pdm is installed using these steps from https://pdm-project.org/latest/usage/advanced/#use-pdm-in-a-multi-stage-dockerfile

When installing like this, this issue appears:

pip install -U pip setuptools wheel
pip install pdm

And the issue disappears if I fix the unearth version:

pip install -U pip setuptools wheel
pip install pdm unearth==0.12.1

Additional context Will try to provide more details if necessary. It's a bit difficult to provide an MRE in this case as there any many components at play. In fact, this issue appears inside a Docker build job running inside a Jenkins pipeline.

[MikulasZelinka]

Looks like PDM simply doesn't pass the indexes to unearth?

https://github.com/pdm-project/pdm/blob/7abdbcc1d33251f7f8dc06bd7d8cb8e56e78028e/src/pdm/environments/base.py#L154C5-L166

Let me know if I should make an issue in PDM instead.

A quick fix would probably be to specify ==0.12.1 here: https://github.com/pdm-project/pdm/blob/7abdbcc1d33251f7f8dc06bd7d8cb8e56e78028e/pyproject.toml#L21

[MikulasZelinka]

Possibly fixed by https://github.com/pdm-project/pdm/pull/2563