pip-tools

pip-tools copied to clipboard

When using pip-compile with --generate-hashes against a private Nexus repository (setup as PyPI proxy) only 2 hashes are generated for multidict==4.7.6 while running pip-compile against pypi.org provides 16 hashes.

This missing hashes lead to issues when building on a different platform.

At first glance it appears the following is happening:

• pip-compile skips dependencies not matching the current platform as packages links are being analyzed.
• pypi.org implements the json api and in the json response has all hashes which are then captured.

My understanding is that this is as it should be.

However Nexus doesn't implement the json API [NEXUS-22828] Support PyPI JSON API - Sonatype JIRA and as a consequence only the artifacts for links not skipped are available locally to compute hash values.

I am filing this issue to understand whether it is expected that repositories implement the json API for pip-compile to work in this scenario.

Environment Versions

1. OS Type: macOS 10.15.7
2. Python version: Python 3.8.1
3. pip version: 21.1.2
4. pip-tools version: 6.1.0

pip-compile --version results in importlib.metadata.PackageNotFoundError: piptools. It was installed as follows:

$ python -m venv venv
$ source venv/bin/activate
$ pip install pip-tools==6.1.0

Steps to replicate

1. See note above for installation
2. Create file requirements.in containing multidict==4.7.6
3. Ensure that there is no requirements.txt
4. Run pip-compile -v -v -r --generate-hashes requirements.in 2> log-with-pypi.txt
5. Remove requirements.txt
6. Have environment variables PIP_INDEX_URL, PIP_TRUSTED_HOST and ~/.netrc point to private nexus.
7. Run pip-compile -v -v -r --generate-hashes requirements.in 2> log-with-nexus.txt

Expected result

The Nexus generated hashes should be the same as the one for PyPI.

Actual result

For pypi.org requirement.txt file content:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes requirements.in
#
multidict==4.7.6 \
    --hash=sha256:1ece5a3369835c20ed57adadc663400b5525904e53bae59ec854a5d36b39b21a \
    --hash=sha256:275ca32383bc5d1894b6975bb4ca6a7ff16ab76fa622967625baeebcf8079000 \
    --hash=sha256:3750f2205b800aac4bb03b5ae48025a64e474d2c6cc79547988ba1d4122a09e2 \
    --hash=sha256:4538273208e7294b2659b1602490f4ed3ab1c8cf9dbdd817e0e9db8e64be2507 \
    --hash=sha256:5141c13374e6b25fe6bf092052ab55c0c03d21bd66c94a0e3ae371d3e4d865a5 \
    --hash=sha256:51a4d210404ac61d32dada00a50ea7ba412e6ea945bbe992e4d7a595276d2ec7 \
    --hash=sha256:5cf311a0f5ef80fe73e4f4c0f0998ec08f954a6ec72b746f3c179e37de1d210d \
    --hash=sha256:6513728873f4326999429a8b00fc7ceddb2509b01d5fd3f3be7881a257b8d463 \
    --hash=sha256:7388d2ef3c55a8ba80da62ecfafa06a1c097c18032a501ffd4cabbc52d7f2b19 \
    --hash=sha256:9456e90649005ad40558f4cf51dbb842e32807df75146c6d940b6f5abb4a78f3 \
    --hash=sha256:c026fe9a05130e44157b98fea3ab12969e5b60691a276150db9eda71710cd10b \
    --hash=sha256:d14842362ed4cf63751648e7672f7174c9818459d169231d03c56e84daf90b7c \
    --hash=sha256:e0d072ae0f2a179c375f67e3da300b47e1a83293c554450b29c900e50afaae87 \
    --hash=sha256:f07acae137b71af3bb548bd8da720956a3bc9f9a0b87733e0899226a2317aeb7 \
    --hash=sha256:fbb77a75e529021e7c4a8d4e823d88ef4d23674a202be4f5addffc72cbb91430 \
    --hash=sha256:fcfbb44c59af3f8ea984de67ec7c306f618a3ec771c2843804069917a8f2e255 \
    --hash=sha256:feed85993dbdb1dbc29102f50bca65bdc68f2c0c8d352468c25b54874f23c39d
    # via -r requirements.in

While for the the private nexus the generated hashes are:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --generate-hashes requirements.in
#
--index-url https://nexus.redacted.com/repository/pypi-all/simple
--trusted-host nexus.redacted.com

multidict==4.7.6 \
    --hash=sha256:c026fe9a05130e44157b98fea3ab12969e5b60691a276150db9eda71710cd10b \
    --hash=sha256:fbb77a75e529021e7c4a8d4e823d88ef4d23674a202be4f5addffc72cbb91430
    # via -r requirements.in

See also log files collected log-with-pypi.txt log-with-nexus.txt

I ran into this too, although using Artifactory as it too doesn't provide the JSON API.

A workaround to avoid piptools hashes files locally is to get Artifactory (and perhaps Nexus can do the same, I've not used it?) to also proxy the API endpoint as a seperate remote repository, and pass this using the --extra-index-url=. This lookup fails for the package, but will succeed for the hashes API.

I also had trouble with this bug. Hash discovery used to work fine with pip-tools==4.4.0 and pip==20.0.2, but broke after I upgraded to pip-tools==6.3.0 and pip==21.2.4.

I stumbled across a crazy workaround:

1. delete requirements.txt
2. pip-compile --generate-hashes requirements.in -- only two hashes show up (source package + current platform)
3. pip-compile --generate-hashes --no-reuse-hashes requirements.in -- all the missing hashes are added.

Note that directly using --no-reuse-hashes in step 2 does not help (I still only get two hashes), but if I run pip-compile twice this works as a workaround.

The tracking issue for the bug is here #1536. Also, there is a WIP PR, reviews and tests are welcome. I'll close this then as a duplicate.