pipenv with production-only dependencies

[dmillard]

Similar to issue #2729, I am also using tensorflow. Unfortunately, my dev and prod setups both run linux and are separated effectively only by the presence of GPUs. Having read PEP 508, I don't believe there is enough information in the markers language to distinguish these systems. Is there some other way that future versions of pipenv can select dependencies? Perhaps through a prod-only section, or similar.

e.g.

...
[packages]
foo = "*"

[prod-packages]
tensorflow-gpu = "*"

[dev-packages]
tensorflow = "*"
...

where [prod-packages] are only installed if --dev is missing?

Originally posted by @dmillard in https://github.com/pypa/pipenv/issues/2729#issuecomment-426514834

[ghost]

Consider extracting gpu related parts into separate project and add dependency. In that way you can reproduce your production setup on your dev setup for at least base package without unneeded workarounds.

[dmillard]

@maln0ir I don't understand how this fixes the issue, since I have the same Pipfile for both dev and prod. If you can provide an example of what you mean, maybe that will help me to understand your suggestion.

[josephong]

@dmillard Did you find a solution for this? This is similar to https://github.com/pypa/pipenv/issues/2729. (Also I just noticed this was posted at the end of that thread).

We're doing some development in an application A that uses a package X that relies on either tensorflow or tensorflow-gpu.

Package X specifies a gpu or non-gpu installations of tensorflow through extras_require in setup.py. The project's setup instructions asks developments to use certain environment variables to determine whether to run pip install package[gpu] or pip install package[non-gpu].

We'd love to migrate to Pipfile for local development for application A (and others), but there's no declarative way to express checking environment variables in the Pipfile. Some people work on multiple machines, some of which have GPUs and others which don't. Markers sound like a good candidate, but I don't think PEP508 specifies a way to inspect environment variables as markers, or to natively detect things like GPU support, which would totally solve our issue. Is PEP508 still a candidate for expansion or is it totally set (@techalchemy I think you had mentioned something about this)?

Supporting different [xyz-packages] sections could also solve this, or multiple Pipfiles. This is sort of like having requirements.$PLATFORM.txt, which is admittedly super nasty, like using sledgehammer to crack a nut. Expanding PEP508 seems nicer.

(P.S. TBH I totally get that this seems like more of a tensorflow issue. In fact for tensorflow, one may be able to circumvent the dependency nastiness – only install tensorflow-gpu and programmatically force it to run on CPUs, but this is not well documented. Other examples of overlapping import namespaces are psycopg2 and psycopg2-binary.)

[mangelozzi]

I have a use case where the production setup uses uwsgi, which is not available for windows (my dev setup). Every time I install a new package on my dev machine I get this error because uwsgi is not available for windows:

pipenv install django-debug-toolbar --dev --skip-lock
Installing django-debug-toolbar…
Adding django-debug-toolbar to Pipfile's [dev-packages]…
Installation Succeeded
Installing dependencies from Pipfile…

An error occurred while installing uwsgi! Will try again.
  ================================ 14/14 - 00:00:08
Installing initially failed dependencies…
[pipenv.exceptions.InstallError]:   File "c:\users\michael\appdata\local\programs\python\python37\lib\site-packages\pipenv\core.py", line 1992, in do_install
[pipenv.exceptions.InstallError]:       skip_lock=skip_lock,
[pipenv.exceptions.InstallError]:   File "c:\users\michael\appdata\local\programs\python\python37\lib\site-packages\pipenv\core.py", line 1253, in do_init
[pipenv.exceptions.InstallError]:       pypi_mirror=pypi_mirror,
[pipenv.exceptions.InstallError]:   File "c:\users\michael\appdata\local\programs\python\python37\lib\site-packages\pipenv\core.py", line 859, in do_install_dependencies
[pipenv.exceptions.InstallError]:       retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
[pipenv.exceptions.InstallError]:   File "c:\users\michael\appdata\local\programs\python\python37\lib\site-packages\pipenv\core.py", line 763, in batch_install
[pipenv.exceptions.InstallError]:       _cleanup_procs(procs, not blocking, failed_deps_queue, retry=retry)
[pipenv.exceptions.InstallError]:   File "c:\users\michael\appdata\local\programs\python\python37\lib\site-packages\pipenv\core.py", line 681, in _cleanup_procs
[pipenv.exceptions.InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: ['Collecting uwsgi', '  Using cached https://files.pythonhosted.org/packages/e7/1e/3dcca007f974fe4eb369bf1b8629d5e342bb3055e2001b2e5340aaefae7a/uwsgi-2.0.18.tar.gz']
[pipenv.exceptions.InstallError]: ['ERROR: Command errored out with exit status 1:', '     command: \'c:\\users\\michael\\.virtualenvs\\project-cdrggqjy\\scripts\\python.exe\' -c \'import sys, setuptools, tokenize; sys.argv[0] = \'"\'"\'C:\\\\Users\\\\Michael\\\\AppData\\\\Local\\\\Temp\\\\pip-install-obyvf2lq\\\\uwsgi\\\\setup.py\'"\'"\'; __file__=\'"\'"\'C:\\\\Users\\\\Michael\\\\AppData\\\\Local\\\\Temp\\\\pip-install-obyvf2lq\\\\uwsgi\\\\setup.py\'"\'"\';f=getattr(tokenize, \'"\'"\'open\'"\'"\', open)(__file__);code=f.read().replace(\'"\'"\'\\r\\n\'"\'"\', \'"\'"\'\\n\'"\'"\');f.close();exec(compile(code, __file__, \'"\'"\'exec\'"\'"\'))\' egg_info --egg-base pip-egg-info', '         cwd: C:\\Users\\Michael\\AppData\\Local\\Temp\\pip-install-obyvf2lq\\uwsgi\\', '    Complete output (7 lines):', '    Traceback (most recent call last):', '      File "<string>", line 1, in <module>', '      File "C:\\Users\\Michael\\AppData\\Local\\Temp\\pip-install-obyvf2lq\\uwsgi\\setup.py", line 3, in <module>', '        import uwsgiconfig as uc', '      File "C:\\Users\\Michael\\AppData\\Local\\Temp\\pip-install-obyvf2lq\\uwsgi\\uwsgiconfig.py", line 8, in <module>', '        uwsgi_os = os.uname()[0]', "    AttributeError: module 'os' has no attribute 'uname'", '    ----------------------------------------', 'ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.']
ERROR: ERROR: Package installation failed...
     ================================ 0/1 - 00:00:04

I hence I have never had an error free installation using pipenv on my dev computer.

EDIT: My situation was solved by using markers. However one could have a linux dev env which one doesnt want to install uwsgi.

[frostming]

@michael-angelozzi Write your Pipfile as below:

# ...
uwsgi = {version="*", sys_platform="!='win32'"}

Then uwsgi will be skipped on Windows

[matteius]

I think this will be supported better by two things: 1.) Category based package groups -- this is in progress with #4745 2.) Transitive dependencies for other platforms than the system being locked on is not supported by the pip resolver and after spending a bunch of time on it, I realized you can add these specific examples to the Pipfile with the appropriate platform marker and then the transitive dependency will be locked on alternate platforms.

Is this issue supposed to cover anything other than that? I gave it a reading, but trying to get the backlog cleaned up before Haccktoberfest to make it more clear where folks should spend time.