pip-tools
pip-tools copied to clipboard
Get hashes from PyPI JSON API fails when index URL ends with slash
When using https://pypi.org/simple/
instead of https://pypi.org/simple
as index URL, pip-compile falls back to hashing files.
This is caused by passing the index URL to pip's PackageIndex
and then calling package_index.pypi_url
: https://github.com/jazzband/pip-tools/blob/1e05d0028ce16d4b2de53ae38a33df7feb84e800/piptools/repositories/pypi.py#L264-L269
- pip itself only uses
PackageIndex
to initialize the constantsPyPI
andTestPyPI
. I don't know what the recommended way is to convert between Simple API URLs and JSON API URLs. - PEP 503 – Simple Repository API says that “PyPI’s base URL is https://pypi.org/simple/”. I don't see why the trailing slash would be considered wrong when used in an index URL.
- Sure, this shouldn't happen a lot for people who use PyPI.org. I've noticed it because AWS CodeArtifact generates/documents index URLs that end with
/simple/
and thenpackage_index.simple_url
~as used in jazzband/pip-tools#1556~ also doesn't work. Butpackage_index.simple_url
would have been my first approach for a potential fix of jazzband/pip-tools#1135.Code Example
from pip._internal.commands import create_command from pip._internal.models.index import PackageIndex command = create_command("install") options, _ = command.parse_args(["-i", "https://pypi.org/simple/"]) session = command._build_session(options) finder = command._build_package_finder(options=options, session=session) for index_url in finder.search_scope.index_urls: package_index = PackageIndex(url=index_url, file_storage_domain="") print(package_index.simple_url) # prints: https://pypi.org/simple/simple print(package_index.pypi_url) # prints: https://pypi.org/simple/pypi
Environment Versions
- OS Type: Linux
- Python version:
$ python -V
: Python 3.10.6 - pip version:
$ pip --version
: pip 22.2.2 - pip-tools version:
$ pip-compile --version
: pip-compile, version 6.8.0
Steps to replicate
-
echo "click" > requirements.in
-
pip-compile --generate-hashes --verbose --index-url=https://pypi.org/simple/ requirements.in
Expected result
Log output
Using indexes:
https://pypi.org/simple/
ROUND 1
Current constraints:
click (from -r requirements.in (line 1))
Finding the best candidates:
found candidate click==8.1.3 (constraint was <any>)
Finding secondary dependencies:
click==8.1.3 requires -
------------------------------------------------------------
Result of round 1: stable, done
Generating hashes:
click
#
# This file is autogenerated by pip-compile with python 3.10
# To update, run:
#
# pip-compile --generate-hashes --index-url=https://pypi.org/simple/ requirements.in
#
click==8.1.3 \
--hash=sha256:7682dc8afb30297001674575ea00d1814d808d6a36af415a82bd481d37ba7b8e \
--hash=sha256:bb4d8133cb15a609f44e8213d9b391b0809795062913b383c62be0ee95b1db48
# via -r requirements.in
Actual result
Log output
Using indexes:
https://pypi.org/simple/
ROUND 1
Current constraints:
click (from -r requirements.in (line 1))
Finding the best candidates:
found candidate click==8.1.3 (constraint was <any>)
Finding secondary dependencies:
click==8.1.3 requires -
------------------------------------------------------------
Result of round 1: stable, done
Generating hashes:
click
Couldn't get hashes from PyPI, fallback to hashing files
Hashing click-8.1.3-py3-none-any.whl
|████████████████████████████████| 100%
Hashing click-8.1.3.tar.gz
|████████████████████████████████| 100%
#
# This file is autogenerated by pip-compile with python 3.10
# To update, run:
#
# pip-compile --generate-hashes --index-url=https://pypi.org/simple/ requirements.in
#
click==8.1.3 \
--hash=sha256:7682dc8afb30297001674575ea00d1814d808d6a36af415a82bd481d37ba7b8e \
--hash=sha256:bb4d8133cb15a609f44e8213d9b391b0809795062913b383c62be0ee95b1db48
# via -r requirements.in
Thanks for the issue @michael-k. Good catch!