pdm
pdm copied to clipboard
pdm-project
Private Google Cloud PyPi proxy causes unnecessary package downloads
Describe the bug
We use a Google Artifact Registry (GAR) repository as a proxy for the public PyPi repository. We have noticed that adding/updating packages is much slower using the GAR proxy than public PyPi. Inspecting the logging from pdm update -v reveals PDM seems to be redownloading packages when resolving while this does not happen using public PyPi.
Example of such download (full logging below):
unearth.preparer: Downloading <Link https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/pytest/pytest-8.3.3-py3-none-any.whl (from https://<REGION>-python.pkg.dev/artifacts-downloads/namespaces/<PROJECT>/repositories/<REPO>/downloads/<LONG_RANDOM_STRING>)> (342 kB)
To reproduce
Minimal pyproject.toml:
[project]
name = "test-pdm-private-pypi"
requires-python = "==3.12.*"
dependencies = [
"pytest==8.3.3",
"ruff==0.7.2",
]
[tool.pdm]
distribution = false
[[tool.pdm.source]]
name = "pypi"
url = "https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/simple/"
verify_ssl = true
Expected Behavior
PDM should not redownload (cached) packages
Environment Information
PDM version:
2.20.0.post1
Python Interpreter:
/Users/<USER>/workspaces/pdm-test/.venv/bin/python (3.12)
Project Root:
/Users/<USER>/workspaces/pdm-test
Local Packages:
{
"implementation_name": "cpython",
"implementation_version": "3.13.0",
"os_name": "posix",
"platform_machine": "arm64",
"platform_release": "23.6.0",
"platform_system": "Darwin",
"platform_version": "Darwin Kernel Version 23.6.0: Thu Sep 12 23:36:55 PDT 2024; root:xnu-10063.141.1.701.1~1/RELEASE_ARM64_T8112",
"python_full_version": "3.12.7",
"platform_python_implementation": "CPython",
"python_version": "3.12",
"sys_platform": "darwin"
}
pdm -v output
Running pdm update -v --no-sync ruff
With public PyPy:
Updating packages: ruff.
pdm.termui: ======== Start resolving requirements ========
pdm.termui: pytest==8.3.3
pdm.termui: ruff==0.7.2
pdm.termui: Adding requirement python==3.12.*
pdm.termui: Adding requirement pytest==8.3.3
pdm.termui: Adding requirement ruff==0.7.2
pdm.termui: ======== Starting round 0 ========
pdm.termui: Adding new pin: python None
pdm.termui: ======== Starting round 1 ========
pdm.termui: Adding requirement iniconfig(from pytest 8.3.3)
pdm.termui: Adding requirement packaging(from pytest 8.3.3)
pdm.termui: Adding requirement pluggy<2,>=1.5(from pytest 8.3.3)
pdm.termui: Adding requirement colorama; sys_platform == "win32"(from pytest 8.3.3)
pdm.termui: Adding new pin: pytest 8.3.3
pdm.termui: ======== Starting round 2 ========
pdm.termui: Adding new pin: ruff 0.7.2
pdm.termui: ======== Starting round 3 ========
pdm.termui: Adding new pin: pluggy 1.5.0
pdm.termui: ======== Starting round 4 ========
pdm.termui: Adding new pin: colorama 0.4.6
pdm.termui: ======== Starting round 5 ========
pdm.termui: Adding new pin: iniconfig 2.0.0
pdm.termui: ======== Starting round 6 ========
pdm.termui: Adding new pin: packaging 24.2
pdm.termui: ======== Starting round 7 ========
pdm.termui: ======== Resolution Result ========
pdm.termui: python None
pdm.termui: pytest 8.3.3
pdm.termui: ruff 0.7.2
pdm.termui: pluggy 1.5.0
pdm.termui: colorama 0.4.6
pdm.termui: iniconfig 2.0.0
pdm.termui: packaging 24.2
pdm.termui: Fetching hashes for [email protected]
With GAR PyPi proxy:
Updating packages: ruff.
pdm.termui: ======== Start resolving requirements ========
pdm.termui: pytest==8.3.3
pdm.termui: ruff==0.7.2
pdm.termui: Adding requirement python==3.12.*
pdm.termui: Adding requirement pytest==8.3.3
pdm.termui: Adding requirement ruff==0.7.2
pdm.termui: ======== Starting round 0 ========
unearth.preparer: Downloading <Link https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/pytest/pytest-8.3.3-py3-none-any.whl (from https://europe-west4-python.pkg.dev/artifacts-downloads/namespaces/<PROJECT>/repositories/<REPO>/downloads/<RANDOM_STRING>)> (342 kB)
pdm.termui: Adding new pin: python None
pdm.termui: ======== Starting round 1 ========
pdm.termui: Adding requirement iniconfig(from pytest 8.3.3)
pdm.termui: Adding requirement packaging(from pytest 8.3.3)
pdm.termui: Adding requirement pluggy<2,>=1.5(from pytest 8.3.3)
pdm.termui: Adding requirement colorama; sys_platform == "win32"(from pytest 8.3.3)
pdm.termui: Adding new pin: pytest 8.3.3
pdm.termui: ======== Starting round 2 ========
unearth.preparer: Downloading <Link https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/iniconfig/iniconfig-2.0.0-py3-none-any.whl (from https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/simple/iniconfig/)> (5.9 kB)
unearth.preparer: Downloading <Link https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/packaging/packaging-24.2-py3-none-any.whl (from https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/simple/packaging/)> (65 kB)
unearth.preparer: Downloading <Link https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/pluggy/pluggy-1.5.0-py3-none-any.whl (from https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/simple/pluggy/)> (20 kB)
unearth.preparer: Downloading <Link https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/colorama/colorama-0.4.6-py2.py3-none-any.whl (from https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/simple/colorama/)> (25 kB)
pdm.termui: Adding new pin: ruff 0.7.2
pdm.termui: ======== Starting round 3 ========
pdm.termui: Adding new pin: pluggy 1.5.0
pdm.termui: ======== Starting round 4 ========
pdm.termui: Adding new pin: colorama 0.4.6
pdm.termui: ======== Starting round 5 ========
pdm.termui: Adding new pin: iniconfig 2.0.0
pdm.termui: ======== Starting round 6 ========
pdm.termui: Adding new pin: packaging 24.2
pdm.termui: ======== Starting round 7 ========
pdm.termui: ======== Resolution Result ========
pdm.termui: python None
pdm.termui: pytest 8.3.3
pdm.termui: ruff 0.7.2
pdm.termui: pluggy 1.5.0
pdm.termui: colorama 0.4.6
pdm.termui: iniconfig 2.0.0
pdm.termui: packaging 24.2
pdm.termui: Fetching hashes for [email protected]
Additional Context
Note that GAR does a redirect when fetching the HTML with packages versions. For example, https://<REGION>-python.pkg.dev/<PROJECT>/<REPO>/simple/ruff/ redirects to https://<REGION>-python.pkg.dev/artifacts-downloads/namespaces/<PROJECT>/repositories/<REPO>/downloads/<RANDOM_STRING> with some random string at the end of the URL. Might this be interfering with the caching PDM does?
Are you willing to submit a PR to fix this bug?
- [ ] Yes, I would like to submit a PR.
