environment markers for different versions of same package override each other

[anthrotype]

I have a requirements.in file which lists the same package twice, each time using a different environment marker. For python 2.7, I want to pin the version to the last python2-supported one; whereas for python 3 and above, I want to install the current version (which is python3 only).

Now, when I use pip to install from such an abstract requirements.in file, it manages to correctly read the different environment markers, and will install the pinned version on python 2.7 and the latest one on python 3.

However, if I use pip-compile the resulting requirements.txt has wrong version specifier and environment markers.

Here is the content of requirements.in:

$ cat requirements.in
doit==0.29.0 ; python_version=='2.7'
doit>=0.29.0 ; python_version>='3'

And here is the output from pip-compile:

$ pip-compile
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --output-file requirements.txt requirements.in
#
--extra-index-url https://pypi.daltonmaag.com/simple/

cloudpickle==0.2.2        # via doit
doit==0.29.0 ; python_version >= "3"
macfsevents==0.7          # via doit
six==1.10.0               # via doit

You'll notice that doit is pinned to ==0.29.0 but it is followed by the wrong environment marker: it says ; python_version >= "3" but it should have been ; python_version == "2.7". And the current doit version 0.30.3 (as of 29 May 2017) is not listed for python_version >= "3".

The expected output should have been:

cloudpickle==0.2.2        # via doit
doit==0.29.0 ; python_version == "2.7"
doit==0.30.3 ; python_version >= "3"
macfsevents==0.7          # via doit
six==1.10.0               # via doit

I get the same result - doit==0.29.0 ; python_version >= "3" - whether I run pip-compile from python 3.6.1 or from python 2.7.13.

I haven't looked at the code yet, but it looks like the environment marker of the last defined package with a given name overrides previous ones; whereas for the version specifiers, the first one for a given package name holds even if subsequently redefined.

I wonder if there are plans to support this kind of behavior in pip-compile?

Thank you in advance.

Related issues and PRs:

• https://github.com/jazzband/pip-tools/pull/206
• https://github.com/jazzband/pip-tools/pull/460

Environment Versions

1. OS Type: MacOS 10.10.5
2. Python version: 3.6.1
3. pip version: 9.0.1
4. pip-tools version: 1.9.1.dev15+gcd14543

[anthrotype]

There is also another problem: a dependency of a requirement which has an environment marker is always added unconditionally without a marker.

In the example above, the python3-version of doit==0.30.3 does no longer require six==1.10.0, but the latter is still added to the generated requirements.txt (# via doit), so it's always installed even when not required.

I think that the dependencies of install requirements that have environment markers should only be installed when the top-level requirement is. That means, the marker should propagate to all the dependencies of marked requirements (unless these are also required by some other package which has no marker?).

[AndydeCleyre]

@anthrotype

I think the problem demonstrated in your first pip-compile output sample is fixed in the current release; the marker that matches the environment of compilation is now the one in the output.

The second issue, of wanting combined/unified environment compilation, may be better continued at #826, which is really about "combined output" or "merged markers."

The third issue, about "deep" markers, is more directly discussed at #563, but closed by adding documentation. The discussion at #826 includes a hack demonstrating deep markers in a way, which is more feasible now that pip-sync ignores incompatibly marked requirements. I would still like to see a real integration of deep markers somehow, as well.

Can you confirm that the first issue is resolved?

[AndydeCleyre]

I think this issue could use some clarification.

I'll start by including the output generated from the original input, using Python 3 and Linux:

cloudpickle==2.0.0
    # via doit
doit==0.33.1 ; python_version >= "3"
    # via -r requirements.in
pyinotify==0.9.6
    # via doit

1. Only the matching env-markered requirement is used, as expected. If that's not what you want, can you see if #826 is suited to your goals?

2. It also looks like you don't want doit pinned at 0.29.0, although that is a match for the input. Expected behavior here is that if there's no pre-existing output file, a higher version will be used, but if the output exists and includes doit at 0.29.0, it may not be upgraded (unless upgrade flags are used), as it already satisfies the needs specified in the input.

Are you experiencing that with no pre-existing output file?

3. As for the follow-up comment, can you provide a reproduction with current pip-tools?