pip-compile --pre fails to choose correct version

[sambvfx]

trafficstars

pip-compile --pre fails to resolve child dependencies.

Environment Versions

1. OS Type: CentOS Linux release 7.8.2003 (Core)
2. Python version: $ python -V: Python 3.7.9
3. pip version: $ pip --version: pip 22.2.2
4. pip-tools version: $ pip-compile --version: 6.8.0

Steps to replicate

1. Create requirements.in file with only this package specified apache_beam[gcp]==2.39.0
2. Run pip-compile --pre

Expected result

This should pick a suitable version of all child packages and resolve a requirement.txt file.

Actual result

pip-compile is failing to agree on which version of protobuf to use. Many of the dependencies in apache_beam have opinions about which protobuf version range to use. It seems like this issue only occurs with the --pre flag.

[AndydeCleyre]

Thanks!

I reproduced the problem with and without using --pre, with Python 3.10.4 on Pop OS 22.04.

The good news is that it succeeds here when using the new backtracking resolver!

We're hoping to deprecate/ditch the currently-default resolver ASAP, so please try with --resolver=backtracking (or set environment parameter PIP_TOOLS_RESOLVER=backtracking), and let us know if there are any problems from there.

[sambvfx]

Interesting! I'm also now failing to get a resolve without the --pre flag.

Using --resolver=backtracking does work with just that beam package, but with our full set of packages it seems to hang. As I added more and more packages it became slower and slower. With our full set of requirements, I let it run for many hours and providing the -v flag the last thing it prints is ROUND 1.

I can try to gather up all of the packages we're resolving and pass them to you for debugging if you'd like.

[AndydeCleyre]

It might be downloading some huge packages to cache. You might get more interesting info with -vvv.