With pre-releases, pip-compile sometimes adds local version hash, sometimes does not

[tuukkamustonen]

pip-compile sometimes locks to a version with PEP 440 local version identifier, sometimes doesn't.

Environment Versions

1. OS Type: linux/KDE Neon (=Ubuntu 18.04 based)
2. Python version: 3.5.5
3. pip version: 9.0.1 and 19.0.3
4. pip-tools version: 3.6.0

Steps to replicate

This happens with package in our company's internal index (devpi), I'm not sure how to reproduce this with a package in PyPI. But maybe someone has similar experiences...

In this example dep is dependency for mypkg. There are 1.0.0 versions of both, but there's also 1.0.1.dev3 version of dep that I would like to test in conjunction.

Prepare:

$ pyenv virtualenv 3.5.5 foobar
...
$ pyenv activate foobar
...

No problem here:

$ rm -rf ~/.cache/pip-tools/
$ echo "\
dep == 1.0.1.dev3\
"  > requirements.in
$ pip-compile --no-header --no-index
dep == 1.0.1.dev3
$ echo "\
dep == 1.0.1.dev3\
mypkg == 1.0.0\
"  > requirements.in
$ pip-compile --no-header --no-index
dep == 1.0.1.dev3
mypkg == 1.0.0

So, dep==1.0.1.dev3 version always when dep is first pinned, and mypkg is added later.

However, here's the problem:

$ rm -rf ~/.cache/pip-tools/
$ echo "\
dep == 1.0.1.dev3\
mypkg == 1.0.0\
"  > requirements.in
$ pip-compile --no-header --no-index
dep == 1.0.1.dev3+gd7cef3b
mypkg == 1.0.0

So, in the failure scenario, dep gets +gd7cef3b local version, when it's in conjunction with mypkg.

Though, it also happens when mypkg is already resolved and when dep is changed to point to pre-release.

$ rm -rf ~/.cache/pip-tools/
$ echo "\
mypkg == 1.0.0\
"  > requirements.in
$ pip-compile --no-header --no-index
dep == 1.0.0
mypkg == 1.0.0
$ echo "\
dep == 1.0.1.dev3\
mypkg == 1.0.0\
"  > requirements.in
$ pip-compile --no-header --no-index
dep == 1.0.1.dev3+gd7cef3b
mypkg == 1.0.0

So it fails always when the mypkg dependency is present.

The local version is removed if I remove mypkg and only directly depend on dep:

$ echo "\
dep == 1.0.1.dev3\
"  > requirements.in
$ pip-compile --no-header --no-index
dep == 1.0.1.dev3

This only happens with pre-releases (as actual releases don't have local identifiers). I don't think global PyPI even accepts local versions.

[atugushev]

Hello @tuukkamustonen,

Thank you for the report! Could you upload mypkg and dep on the Test PyPI. I mean not your real packages, but test packages like this one. Could you reproduce the bug on that env?

[tuukkamustonen]

I will look into it, but unfortunately I don't have time for that right now. Need to come back to this a bit later.

[tuukkamustonen]

Still happening in 4.1.0. I really need to check this.

[atugushev]

Working on #1114 I've possibly tracked down the issue. pip resolver searches all candidates matched to the version, cut down local version identifier (e.g., "+gd7cef3") and get maximum from the list of cut versions, that's why the result is so arbitrary.

Suppose you have the following installation candidates for foo=1.0.0:

• foo=1.0.0, url=http://index-url/foo-1.0.0.tar.gz
• foo=1.0.0+dev, url=file:///tmp/path/to/package
• foo=1.0.0+b02632, [email protected]:user/foo.git@02632

pip transforms these candidates to install requirements (see https://github.com/jazzband/pip-tools/issues/1114#issuecomment-617167490 for the traceback):

• foo=1.0.0, url=http://index-url/foo-1.0.0.tar.gz
• foo=1.0.0, url=file:///tmp/path/to/package
• foo=1.0.0, [email protected]:user/foo.git@02632

and then gets a package with a max version, thus the result of max() would be arbitrary due to all candidates have the same version.

[atugushev]

It could be a pip bug. I'll try to provide reproducible steps.

[tuukkamustonen]

Just wanted to update: we recently migrated from devpi to Artifactory and the issue persists. So, I would say it's at least not index-specific.

[tuukkamustonen]

Just chiming in that this still happens for me. Lots of things have changed (pip, pip-tools version, some setup.py details, etc.) but the behavior remains the same.

I wonder if I ever get time to get to the bottom of this! 😛

[atugushev]

Could you test it with pip-compile --resolver backtracking?

[tuukkamustonen]

Unfortunately, I've switched to a new employer and don't have the issue producing code/setup at hand anymore 🤷🏻‍♂️

There seems to be little traffic in this ticket, so you might want to close it and wait for someone else to report the same 🤔

[atugushev]

I'll close this based on the above. Thanks!