Pipenv that includes Jupyter can’t be installed

[mjpieters]

Bug description

Pipenv is run with the --system cli switch, so any dependence inside the lock filter that clash with the system packages cause the build to fail.

E.g. https://mybinder.org/v2/gh/mjpieters/adventofcode/b7f80b433311f035916a0bd74fae818b9c231105 fails to install as pip isn’t allowed to uninstall terminado:

[pipenv.exceptions.InstallError]: ['Collecting terminado==0.8.3 (from 
-r /tmp/pipenv-jtgp9b6o-requirements/pipenv-y1rwqxuh-requirement.txt (
line 1))', '  Using cached https://files.pythonhosted.org/packages/ff/
96/1d9a2c23990aea8f8e0b5c3b6627d03196a73771a17a2d9860bbe9823ab6/termin
ado-0.8.3-py2.py3-none-any.whl', 'Installing collected packages: termi
nado', '  Found existing installation: terminado 0.8.2']              
[pipenv.exceptions.InstallError]: ["ERROR: Cannot uninstall 'terminado
'. It is a distutils installed project and thus we cannot accurately d
etermine which files belong to it which would lead to only a partial u
ninstall."]                                                           
ERROR: ERROR: Package installation failed...                          
Removing intermediate container 689fa80c20bf

I’ve picked a specific revision for the above mybinder URL to ensure reproducibility; see the repository itself (same rev).

Perhaps the build should be retried without the --system switch in such cases?

[consideRatio]

I've struggled a lot with these considerations without coming to a great conclusion =/

Without the --system flag, the dependencies are installed within a virtual environment. I think it is very relevant that this virtual environment is systematically entered if we do it like that. We need to enter it when working from a terminal so we have access to python installed executables, and we need to enter it when opening notebooks etc.

I tried accomplishing that and failed... I don't know what I think makes the most sense still. Perhaps we should do it without system as we should really want to install exactly what is specified for reproducibility's sake, but what steps should we then take to make it keep working when the system packages cause conflicts?

Note also btw, that this issue is actually in a way more general issue than specifically for Pipenv's buildpack. It is simply so that using pipenv we end up with more specific dependencies due to the Pipfile.lock file used by default.

[mjpieters]

Would converting the lock file work better? E.g. use https://pypi.org/project/dephell/ or the jq one-liners shown in https://github.com/pypa/pipenv/issues/3493 to extract a requirements.txt file and install that?

[consideRatio]

I don't see how that helps, the issue as I understand it is that the lock file pinned exactly all dependencies, and that gave no wiggle room. We like to have it pinned down exactly in a way though, because why not?

We could use the Pipfile instead of the Pipfile.lock, that would resolve this issue but cause the issue of no longer reproducing the environment as one would presume, and that could cause something initially working to stop working etc.

[mjpieters]

Right, but you can’t have it both ways ;-) Either no locking, or entering the virtualenv with locked-down dependencies.

You can use pipenv run ... to run commands in the venv, in case that helps. Or just use pipenv—venv (I think) to get the venv path so you can inject the necessary environment variables manually or use its bin/activate script.

[y2kbugger]

I tried accomplishing that and failed... I don't know what I think makes the most sense still. Perhaps we should do it without system as we should really want to install exactly what is specified for reproducibility's sake, but what steps should we then take to make it keep working when the system packages cause coflicts?

I think Virtual env makes, can you describe what you mean by "failed" when trying to enter he venv?

[rdbisme]

Hello, can anyone give a test to https://github.com/jupyterhub/repo2docker/pull/989?

[minrk]

This should be considered a metadata bug in the terminado conda package, fixed by https://github.com/conda-forge/terminado-feedstock/pull/16

Making sure terminado is >= 0.9.1 (true for Python >= 3.6) should avoid this. Is this still an issue that can be reproduced?

[rdbisme]

I still have the problem: https://josiepy.rdb.is/examples/001_Getting_Started.html (clicking on the Binder badge).

This is the Pipfile. I'm currently using pipenv for development purposes since I have few packages that are vendored.

Can you maybe help me understanding what I'm doing wrong? Theoretically jupyter should not be included in the dependencies since the setup.py adds it only for the extra examples or all, and the package is installed in editable mode from the Pipfile without those extras being flagged.

This is an excerpt of the log on binder (for whatever reason I can't copy the log on binder)