Poetry artifact cache breaks pip's wheel cache

[maksbotan]

• [x] I am on the latest Poetry version.

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: debian:buster in Docker

• Poetry version: 1.1.3

Issue

Hi,

Starting with some version poetry started to store pypi artifacts in ~/.cache/pypoetry/artifacts, including wheels and source .tar.gz distributions when there are no wheels on pypi.

Later poetry calls pip install with an absolute path to those downloaded artifacts. However, in this mode pip does not look into its own cache for already built wheels and always rebuilds from source (it does store the resulting wheel in cache though).

When the package in question contains C extensions, for example, this behavior causes long build times. In my particular scenario, I run poetry install a lot in CI jobs and I'd like to benefit from caching the already built wheels.

Is there anything that can be done to mitigate this?

[1ace]

I second that; it's extra annoying in a CI where you run the same poetry install thousands of times and you have to recompile the same packages every time instead of installing the one that's right there in the cache.

[1ace]

I've been digging through the code, and it's a bit unclear to me if poetry is supposed to do something differently, or if pip is the one who's supposed to figure out it needs to check its cache before re-building a wheel?

I think it's the latter, in which case this issue can be closed and one should be open in https://github.com/pypa/pip/issues (if there isn't already one, but I haven't found one)

@sdispater any thoughts?

[1ace]

Something like this might do it:

diff --git a/poetry/installation/pip_installer.py b/poetry/installation/pip_installer.py
index df1249737a1fe21b9122..ff54ac6d8589e16a58f3 100644
--- a/poetry/installation/pip_installer.py
+++ b/poetry/installation/pip_installer.py
@@ -6,6 +6,11 @@
 
 from clikit.api.io import IO
 
+try:
+    from pip._internal.utils.appdirs import user_cache_dir
+except ImportError:  # pip < 10
+    from pip.utils.appdirs import user_cache_dir
+
 from poetry.core.pyproject.toml import PyProjectTOML
 from poetry.repositories.pool import Pool
 from poetry.utils._compat import encode
@@ -34,6 +39,6 @@ def install(self, package, update=False):
 
-        args = ["install", "--no-deps"]
+        args = ["install", "--no-deps", "--find-links", user_cache_dir('wheel')]
 
         if (
             package.source_type not in {"git", "directory", "file", "url"}
             and package.source_url

[Aloisius]

#3732 should fix the issue. The problem is that pip will not check its wheel cache of previously built packages unless you pass in a requirements.txt file since the logic for it is in the link resolver. Poetry simply does a pip install --no-deps cache/artifacts/.../package-1.0.0.tar.gz which bypasses it.

The patch creates a temporary requirements.txt file and does a pip install --no-deps -r temp_requirements.txt on it. It seems to solve the problem for me, but if someone could test it to be sure.

$ poetry install
$ rm -rfi `poetry env info -p`
$ export PIP_LOG=$PWD/pip.log
$ poetry install

That should show it installing wheels it had previously built in the pip.log file.

[ironhalik]

Any news regarding this issue? Crypto is moving away from proof-of-work, and so could we stop building wheels constantly :)

[dimbleby]

#6205 renders pip's cache irrelevant