pip-tools icon indicating copy to clipboard operation
pip-tools copied to clipboard

`@ git+` requirements with backtracking resolver can result in pins to local wheel cache

Open gschaffner opened this issue 2 years ago • 1 comments

If an @ git+ requirement is present in pip's wheel cache, pip-compile --resolver backtracking will pin to @ file://....whl.

(Moved from https://github.com/jazzband/pip-tools/pull/1539#issuecomment-1170845933. @ git+ pinned package switched from pip-tools to typeguard for clarity.)

Environment Versions

  1. OS Type: Manjaro 21.3.1
  2. Python version: CPython 3.10.5
  3. pip version: 22.1.2
  4. pip-tools version: 1e05d0028ce16d4b2de53ae38a33df7feb84e800

Steps to replicate

#!/usr/bin/env bash

set -Eeuo pipefail
shopt -s inherit_errexit

cd "$(mktemp -d)"

echo 'typeguard @ git+https://github.com/agronholm/typeguard.git@a298d6cde249fe7ad46b90b94aa3bdd246e60417' > requirements.in
# cache ~/.cache/pip/wheels/a3/ac/4a/1aa2eba1bf7a5f0259fbc2f4160621fe8ea72613653e32c6fe/typeguard-2.13.3.post48-py3-none-any.whl
python -m venv --upgrade-deps .venv && . .venv/bin/activate
mv ~/.cache/pip/wheels{,.bak}
pip install typeguard==2.13.3
pip install -r requirements.in
# pip-compile
rm -r .venv
python -m venv --upgrade-deps .venv && . .venv/bin/activate
pip install 'pip-tools @ git+https://github.com/jazzband/pip-tools.git@1e05d0028ce16d4b2de53ae38a33df7feb84e800'
pip-compile --resolver backtracking
# restore cache
rm -r ~/.cache/pip/wheels
mv ~/.cache/pip/wheels{.bak,}

Expected result

typeguard @ git+https://github.com/agronholm/typeguard.git@a298d6cde249fe7ad46b90b94aa3bdd246e60417

Actual result

< ... truncated for brevity ... >
typeguard @ file:///home/ganden/.cache/pip/wheels/a3/ac/4a/1aa2eba1bf7a5f0259fbc2f4160621fe8ea72613653e32c6fe/typeguard-2.13.3.post48-py3-none-any.whl
    # via -r requirements.in

gschaffner avatar Jun 30 '22 07:06 gschaffner

At the first glance these might be the fixes:

  1. Override cached link by source link
diff --git piptools/resolver.py piptools/resolver.py
index f9d3d12..72c95d5 100644
--- piptools/resolver.py
+++ piptools/resolver.py
@@ -763,7 +763,10 @@ def _get_install_requirement_from_candidate(

         # Prepare pinned install requirement. Copy it from candidate's install
         # requirement so that it could be mutated later.
-        pinned_ireq = copy_install_requirement(ireq)
+        kwargs = {}
+        if candidate.source_link is not None:
+            kwargs['link'] = candidate.source_link
+        pinned_ireq = copy_install_requirement(ireq, **kwargs)

         # Canonicalize name
         assert ireq.name is not None
  1. Hack writer
diff --git piptools/utils.py piptools/utils.py
index 47f3637..c7624dd 100644
--- piptools/utils.py
+++ piptools/utils.py
@@ -119,7 +119,9 @@ def format_requirement(
     in a less verbose way than using its `__str__` method.
     """
     if ireq.editable:
-        line = f"-e {ireq.link.url}"
+        req = ireq.req
+        url = req.url if req and req.url else ireq.link.url
+        line = f"-e {url}"
     elif is_url_requirement(ireq):
         line = _build_direct_reference_best_efforts(ireq)
     else:

However, need some research on how InstallRequirement.link and caches work.

atugushev avatar Jun 30 '22 08:06 atugushev