pip-tools icon indicating copy to clipboard operation
pip-tools copied to clipboard

Default to --no-reuse-hashes

Open graingert opened this issue 5 years ago • 10 comments

A breaking change for v6

Changelog-friendly one-liner: default to --no-reuse-hashes

Contributor checklist
  • [ ] Provided the tests for the changes.
  • [ ] Gave a clear one-line description in the PR (that the maintainers can add to CHANGELOG.md on release).
  • [ ] Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

Now that hashing is way faster than it once was, I think the slight speed improvement from caching no longer outweighs the danger of trusting local/accidental changes to the .txt pins

graingert avatar Jul 20 '20 12:07 graingert

Codecov Report

Merging #1184 (5decf31) into master (6ed9301) will not change coverage. The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master    #1184   +/-   ##
=======================================
  Coverage   99.67%   99.67%           
=======================================
  Files          34       34           
  Lines        3042     3042           
  Branches      327      327           
=======================================
  Hits         3032     3032           
  Misses          5        5           
  Partials        5        5           
Impacted Files Coverage Δ
piptools/scripts/compile.py 100.00% <ø> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 6ed9301...5decf31. Read the comment docs.

codecov[bot] avatar Jul 20 '20 13:07 codecov[bot]

Now that hashing is way faster than it once was, I think the slight speed improvement from caching no longer outweighs the danger of trusting local/accidental changes to the .txt pins

If this is the case, would it make sense to deprecate/remove the feature entirely?

jdufresne avatar Nov 21 '20 23:11 jdufresne

@jdufresne I'm in favour of that

graingert avatar Nov 21 '20 23:11 graingert

@jdufresne https://github.com/jazzband/pip-tools/issues/1135 would probably need doing first though

graingert avatar Nov 21 '20 23:11 graingert

FTR: I've removed this PR from 6.0.0. I'm going to prepare a release soon and it's unclear to me whether there should be changed the default value for --no-reuse-hashes or should be completely removed. Seems like this PR could be postponed for now.

atugushev avatar Mar 11 '21 14:03 atugushev

Artifactory doesn't support the /json endpoint, so users in a corporate environment that don't directly pull from PyPI would be required to always download each file to be hashed. So removing the capability to reuse hashes would increase the time it takes to compile a lock file. This is part of the reason I implemented single hash (#1406).

plannigan avatar Jun 09 '21 12:06 plannigan

@plannigan https://github.com/jazzband/pip-tools/issues/1135 we could pull the hashes from the simple API

graingert avatar Jun 09 '21 13:06 graingert

Good to know. If sequence was try /json/try simple/download file then that would resolve my concerns.

plannigan avatar Jun 09 '21 13:06 plannigan

Probably not worth doing the /json endpoint now as the simple api will already have been downloaded with the hashes in

graingert avatar Jun 09 '21 13:06 graingert

There are widely used Python projects that rely extensively on --find-links / --extra-index-url for distribution that don't provide checksums in their index files. For example:

  • pytorch - https://download.pytorch.org/whl/torch/
  • pytorch-scatter - https://data.pyg.org/whl/torch-1.12.0+cpu.html

pytorch is a ~2GB download. Until these projects catch up, generating the hashes will be too expensive (slow and bandwidth intensive) if --reuse-hashes is dropped.

neykov avatar Nov 13 '22 19:11 neykov