different package versions for different python versions

[maparent]

My application can work under either python2 or python3, but some requirements have different versions. (For example, ipython >=6 for python3, < 6 for python2.) If I specify ipython="*", pipenv lock will pick ipython==6.3.1 I tried specifying

ipython = {version = ">=6.3", markers="python_version >= '3'"}
ipython = {version = "<6", markers="python_version < '3'"}

but pipenv complains about re-using the identifier.

I'm not sure if this is possible; if so I could not find it in the documentation (and maybe that case should be made explicit in the documentation.) If it is not possible and is not intended to be, maybe that should also be said explicitly. Regards

[uranusjr]

This is possible (not with this syntax, of course, because TOML does not allow duplicate keys), but I don’t think it’s a good idea overall. It is better to wrap the package (IPython in this example) in a custom setup.py, and specify the conditional dependency there.

Something like:

# ipython-select/setup.py
from setuptools import setup
setup(
    name='ipython-select',
    version='1.0',
    installs_requires=[
        'ipython>=6.3; python_version>="3"',
        'ipython<6; python_version<"3"',
    ],
)

# Pipfile
[packages]
ipython-select = {path = "./ipython-select"}

[maparent]

Thank you, that is helpful.

[kalfa]

Could a similar approach be included in the Pipfile syntax directly? e.g.

[packages]
ipython-select = [{version=">=6.3", markers="python_version>='3'"}, {version="<6", markers="python_version<'3'"}]

or anything like that, not sure if that's TOML compatible either.

otherwise pipenv/pipfile just delegate to setuptools and python code (i.e. setup.py) the expressivity of a project's requirements.

[axnsan12]

Adding onto this - other markers/source combinations are of interest as well, and more difficult to incorporate into a setup.py

For my use case, I would like to install from different wheels on Windows depending on the platform, and from PyPI on Linux; i.e.:

python-ldap = {path = ".\\wheel\\python_ldap-3.1.0-cp36-cp36m-win_amd64.whl", markers="python_version == '3.6' and sys_platform == 'win32' and '64' in platform_machine"}
python-ldap = {path = ".\\wheel\\python_ldap-3.1.0-cp36-cp36m-win32.whl", markers="python_version == '3.6' and sys_platform == 'win32' and '64' not in platform_machine"}
python-ldap = {version  = "==3.1.0", sys_platform = "!= 'win32"}

This is needed in a lot of cases because many package mantainers refuse to provide pre-built wheels for windows via PyPI, forcing users to download them manually from the likes of https://www.lfd.uci.edu/~gohlke/pythonlibs/

It is also easily achievable using pip and normal requirements files.

[uranusjr]

I would encourage people to persue the wrapper package solution. Also, if you wish to provide a solution, please provide one that is actually doable, i.e. at least propose solution in legal TOML syntax. Thanks.

[axnsan12]

How would I solve my problem (install from local wheels on Windows, install from pypi on non-Windows) using a wrapper package?

[kalfa]

@uranusjr can I know why this encouragement? It seems to me that a wrapper solution defeat the purpose of a Pipfile itself.

Why should someone who subscribes to the idea behind pipfile and pipenv, should then choose to use code to describe dependancy as a preferred/encouraged approach? Let's do "that" until is possible otherwise: yes. But encouraging it, seems opposite to the purpose of the project. Am I missing your point?

I agree on the "if propose, at least propose doable" Although, use cases collection can be done with meta code, and doesn't really change anything, while enriching the discussion.

[uranusjr]

The reasoning is a practical one—the feature is highly unlikely to land now, or even in short-term future. While a spec change is indeed worth persueing, it is by nature quite involved, and require a lot of discussion. Assuming you already need the feature right now, it is more preferrable (for us maintainers, at least) to adopt Pipenv now with a workaround, than to put off adapting it altogether until the feature is solidified and land in the far, far future.

My recommendation is also not a discouragement to spec discussion. The spec modification is still the ultimate goal, and I would love to see proposals (if doable), as mentioned above. In the meantime, however, we still need to work on our real-world tasks, and I feel it is the best to solve the problem at hand with a workaround, instead of going directly for the theoratically best, purest solution (which, from my experience, tends to result in hast discussion and bad design decisions).

[yili1992]

@axnsan12 now, Did you have resolution for your problem

[axnsan12]

Nope, I gave up and went back to pip

[maparent]

For the record: after a few experiments, I also decided against using this workaround and went to a pip-tool based solution. I specify python_version in the requirements.in file and generate separate requirements[2,3].txt files. I'm afraid that the need for wrappers will be an obstacle to pipenv adoption. Best of luck.

[uranusjr]

Hey, nobody is stopping you if you want to actually do something about it.

[thernstig]

I unfortunately had to go back to pip as well for a project. Understandable this requires some effort and is in low priority. Hopefully some hero steps in and tries to push a PR.

[foozmeat]

We develop on macos and deploy custom wheels to raspberry pis. A sample from our production requirements.txt is

pandas==0.22.0; sys_platform == 'darwin'
https://our.server/raspbian/pandas-0.22.0-cp36-cp36m-linux_armv7l.whl; sys_platform == 'linux'

This works as expected and requires no accessory files, scripts or workarounds.

OPINION: I really appreciate the work that's been put into pipenv and I use it for several other projects but it simply falls short in this real-world scenario. I feel its disingenuous to demand that people create hacky workarounds because duplicate keys aren't supported by TOML. This case is an argument for a DSL for the Pipfile. It seems silly to hold back pipenv's growth because TOML is insufficient to represent real-world use cases.

✌️

[thernstig]

There are multiple ways it can be achieved with TOML. Which one is the best can be debated. Here is a quick solution:

ipython = [ {version = ">=6.3", markers="python_version >= '3'"},
            {version = "<6", markers="python_version < '3'"} ]

This is a normal array with inline tables, taken directly from the TOML spec.

[ghost]

@uranusjr This seems like a very reasonable request. Do you mind providing us with a status update? I fail to reason that this would be hard to implement. Although, I'm currently busy with other projects at the moment. If I ever get unbusy, maybe I'll send a PR for it.

To be clearly respectful, I removed my response to @techalchemy. I do agree with him after all. Thanks again for the response!

[techalchemy]

@pro-src you may not see why it would be hard to implement, but it touches our dependency resolver which inherently makes it hard to implement. It also requires specification changes to Pipfile which adds complexity. Put another way, if it were easy, it would already have been done.

I did have a chance to chat with @ncoghlan about this and he suggested simply adding the version tag to the lockfile as an extension, we could then modify the lockfile search function to look for lockfiles with extensions first, and, during resolution, if we are resolving with python 2 for instance, we can simply create Pipfile.py27.lock instead of Pipfile.lock. This solution seems pretty elegant to me, albeit a little similar to the current situation with requirements.txt files.

[techalchemy]

no worries, definitely happy to accept a PR and glad you are willing to tackle it :D also glad to know you are looking at the code and not struggling too much, been putting some work into cleaning things up a bit

first step is definitely going to be coming to some agreement about the implementation though, I think that just means at the end of the day that @ncoghlan, @uranusjr and myself should be on the same page and then we should get a markdown document together describing the behavior. Once we get that far it's all just implementation

[thernstig]

@techalchemy Would that approach solve situations like reported here https://github.com/pypa/pipenv/issues/2756, where you want to check which sys_platform is being run?

E.g.

[packages]
bsddb3 = {version = "==6.2.6", sys_platform = "!= 'win32'"}
bsddb3 = {file = "https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl", sys_platform = "== 'win32'", platform_machine = "== 'AMD64'"}

[frostming]

@thernstig Unfortunately duplicate keys are not acceptable in a TOML document, however, you can always modify the file entry to another name, where the package name here doesn't matter if you have specified the installation link or file path.

[uranusjr]

I’ve been wondering whether it would be a good idea to detach entry keys from package names:

[packages]
bsddb3 = { version = "==6.2.6", sys_platform = "!= 'win32'" }
# Defaults to the key when "name" is missing; backwards compatible.

bsddb3-1- = { name = "bsddb3", file = "https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl", sys_platform = "== 'win32'", platform_machine = "== 'AMD64'"}
# The trailing dash is intentional to prevent conflicts
# when there actually is a package named "bsddb3-1".

The resolver will still need work to actually support duplicated package names. This is only the very first (and arguably the easiest) step to make it happen.

[thernstig]

@frostming, but as my previous comment slightly above, you are allowed to do this:

bsddb3 = [ {version = "==6.2.6", sys_platform = "!= 'win32'"},
           {file = "https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl", sys_platform = "== 'win32'", platform_machine = "== 'AMD64'"} ]

Or to make it more clear with another case:

bsddb3 = [ { version = "==7.2.6", sys_platform = "!= 'win32'" },
           { version = "==6.2.6", sys_platform = "== 'win32'" } ]

@uranusjr Seems cool as well.

[ncoghlan]

I quite like the idea of allowing the package requirement to be a list, with the behaviour being that when it's a list, the requirements for all applicable environments are applied. The no-list case is then just a simplification that allows the square brackets to be omitted in the single element list case (i.e. [{...}] -> {...}).

That way it's crystal clear that the requirements all relate to the same package, there are just platform-specific considerations as to exactly which versions you can and can't use. As another practical example, consider the situation where a particular release of a dependency is known to have issues on Windows, but works fine elsewhere, and later versions were fixed to work again on Windows:

example_dependency = [
    {version = ">=7.2""},
    {version = "!=7.3.0", sys_platform = "== 'win32'" }
]

If multiple options in the list end up being applicable, that's fine, but if they generate a conflict, then the dependency resolution will fail, just as it would if the conflicting dependencies had come from different projects.

[ALERTua]

it took me seven centuries to find this issue. just wanted to say that, unfortunately, this issue holds me from moving from pip to pipenv. I would appreciate any workaround or any kind of implementation.

[leesei]

@frostming, but as my previous comment slightly above, you are allowed to do this:

bsddb3 = [ {version = "==6.2.6", sys_platform = "!= 'win32'"},
           {file = "https://download.lfd.uci.edu/pythonlibs/l8ulg3xw/bsddb3-6.2.6-cp36-cp36m-win_amd64.whl", sys_platform = "== 'win32'", platform_machine = "== 'AMD64'"} ]

Or to make it more clear with another case:

bsddb3 = [ { version = "==7.2.6", sys_platform = "!= 'win32'" },
           { version = "==6.2.6", sys_platform = "== 'win32'" } ]

@uranusjr Seems cool as well.

My programmer instinct pointed me to the same solution. To my suprise it lead to an error:

ipfile.lock (44462f) out of date, updating to (c9fed2)...
Traceback (most recent call last):
  File "/usr/bin/pipenv", line 33, in <module>
    sys.exit(load_entry_point('pipenv==2021.5.29', 'console_scripts', 'pipenv')())
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/click/decorators.py", line 73, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/pipenv/vendor/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3.9/site-packages/pipenv/cli/command.py", line 233, in install
    retcode = do_install(
  File "/usr/lib/python3.9/site-packages/pipenv/core.py", line 2052, in do_install
    do_init(
  File "/usr/lib/python3.9/site-packages/pipenv/core.py", line 1274, in do_init
    do_lock(
  File "/usr/lib/python3.9/site-packages/pipenv/core.py", line 1081, in do_lock
    dev_packages = overwrite_dev(project.packages, dev_packages)
  File "/usr/lib/python3.9/site-packages/pipenv/project.py", line 717, in packages
    return self._build_package_list("packages")
  File "/usr/lib/python3.9/site-packages/pipenv/project.py", line 222, in _build_package_list
    or any(is_valid_url(i) for i in [k, v])
  File "/usr/lib/python3.9/site-packages/pipenv/project.py", line 222, in <genexpr>
    or any(is_valid_url(i) for i in [k, v])
  File "/usr/lib/python3.9/site-packages/pipenv/utils.py", line 1745, in is_valid_url
    pieces = urlparse(url)
  File "/usr/lib/python3.9/urllib/parse.py", line 392, in urlparse
    url, scheme, _coerce_result = _coerce_args(url, scheme)
  File "/usr/lib/python3.9/urllib/parse.py", line 128, in _coerce_args
    return _decode_args(args) + (_encode_result,)
  File "/usr/lib/python3.9/urllib/parse.py", line 112, in _decode_args
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
  File "/usr/lib/python3.9/urllib/parse.py", line 112, in <genexpr>
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
AttributeError: 'Array' object has no attribute 'decode'

[packages]
tables = [{version = "*", sys_platform = "!= 'win32'"},{path = "./vendor/tables-3.6.1-cp39-cp39-win_amd64.whl", sys_platform = "== 'win32'"}]

I wonder if the array syntax is just a suggestion to new syntax or an actual solution.

[uranusjr]

I believe we are still waiting for someone to actually implement the array syntax.

[se7enXF]

Same issue. Python 3.7 includes the package contexvars, but Python3.6 does not. I want to install contexvars in setup.py when using Python3.6, but not in Python3.7. How do I make it happen？

[matteius]

Named categories could perhaps solve this problem too #4745 you could have your default group that is common to all systems, and then an ldap-win ldap-linux for example -- it does mean needing to know which package group you want to install ahead of time for that platform, but it feels like a reasonable workaround.

[MitchMTR]

you have reached the end of the internet.

[thernstig]

@matteius no chance this could be re-opended and reconsidered, as this https://github.com/pypa/pipenv/issues/2171#issuecomment-493098208 seems like a nicer solution than https://github.com/pypa/pipenv/issues/4745.

It would bring some extra, nice power.