Does `pip-compile` take a long time to run for you?

[davidleejy]

Hello fellow pip-tool users,

I don't know if it's just me or does pip-compile take a long time to complete? I'm new to pip-tools, love the concept, but am worried that I may be running afoul of best practices.

At the moment what i'm doing repeatedly is running pip-compile -U --resolver=backtracking after editing requirements.in.

A pretty short & basic requirements.in like the following takes pip-compile 3-5 minutes:

# A pretty short & basic requirements.in
bert_score
blobfile
nltk
numpy<1.24.0
packaging
psutil
PyYAML
setuptools
spacy
torch
torchmetrics
tqdm
transformers
wandb
datasets

My intuition tells me that there's a 50% chance that i'm doing something wrong here and I would be grateful for advice 😄

[atugushev]

How much time does pip-compile take with legacy resolver? Also how fast it is without -U flag?

[IzakMaraisTAL]

pip-compile has started taking extremely long for me (killed after more than an hour).

python version: 3.7.10 pip version: 23.0 pip-compile version: 6.10.0

requirements-dev.in:

tfx[kfp]==1.12.0
tensorflow_text
tensorflow_recommenders
scann
tqdm
ujson
google-api-python-client
google-auth
google-auth-httplib2
pytest
pytest-cov
pytest-mock
requests-mock
Pillow
pre-commit

After a bit of debugging, it looks like the -U flag is to blame.

The following completes fast (just over a minute) (output_no_U.txt).

time pip-compile -v --resolver=backtracking requirements-dev.in --output-file requirements-dev.txt | ts -i "{{%H:%M:%.S}}" > output_no_U.txt

Whereas, this does not:

time pip-compile -v -U --resolver=backtracking requirements-dev.in --output-file requirements-dev.txt | ts -i "{{%H:%M:%.S}}" > output_with_U.txt

I usually kill after many minutes (sometimes more than an hour). I have tried multiple times and 9/10 times it would be extremely slow, but one time it was relatively fast at just over 3 minutes. Looking at the attached output_with_U.txt might give some insight into which steps are slow. The incremental timestamps show these steps taking > 40s each:

{{00:00:43.895765}}     Using cached google_cloud_aiplatform-1.16.1-py2.py3-none-any.whl (2.2 MB)
{{00:00:45.071425}}     Using cached google_cloud_aiplatform-1.16.0-py2.py3-none-any.whl (2.2 MB)
{{00:00:49.434197}}     Using cached google_cloud_aiplatform-1.15.1-py2.py3-none-any.whl (2.1 MB)
{{00:00:45.209635}}     Using cached google_cloud_aiplatform-1.15.0-py2.py3-none-any.whl (2.1 MB)
{{00:00:53.008086}}     Using cached google_cloud_aiplatform-1.14.0-py2.py3-none-any.whl (1.9 MB)
{{00:00:52.136853}}     Using cached google_cloud_aiplatform-1.13.1-py2.py3-none-any.whl (1.8 MB)

and some steps taking minutes:

{{00:01:51.449972}}   INFO: pip is looking at multiple versions of absl-py to determine which version is compatible with other requirements. This could take a while.
{{00:00:44.400301}}   INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.

[androiddrew]

This is more of a general question about the difference in the resolvers. I noticed pip-compile --resolver=backtracing is downloading the packages while the legacy resolved is not. I assume that is a "Feature" of the backtracing resolver. I assumed it would be caching what it downloaded but then I see some of the packages being redownloaded during pip-sync.

Can someone explain to me why this is happening? Obviously it is contributing to the time I experience when using the backtracing resolver.

[Shookit]

I just wanted to echo @IzakMaraisTAL -- this is slow enough where it doesn't ever actually resolve on some of my bigger requirements files. I originally thought it might be a pip-tools 6 vs 7 issue, but I confirmed that this is happening on both versions if you explicitly specify --resolver=backtracking.