Multiple python versions used in jupyter environment, causing issues.

[bj0]

Describe the bug

There's a documented way to provide a python version/instance (like from mach-nix) to a kernel environment, but I cannot see any way to do the same for the jupyter environment. This leads to uncertainty about what version of python is being used.

The version ambiguity is somehow leading to multiple version of python being used (3.8 and 3.9), which is causing pyzmq (with a cython backend) to error out.

My previously working shell.nix suddenly started failing from this. I created a simple shell.nix that produces the error:

{ pkgs ? import (fetchTarball https://github.com/nixos/nixpkgs/archive/nixpkgs-unstable.tar.gz) {} }:

let
    jupyter = import (builtins.fetchGit {
        url = https://github.com/tweag/jupyterWith;
    }) {};

    # create jupyter environemnt
    jupyterEnvironment = jupyter.jupyterlabWith {    };

in
    jupyterEnvironment.env

Then, running a jupyter command (like jupyter kernelspec list) is producing the stacktrace:

Traceback (most recent call last): File "/nix/store/slfhacq49zmh8p2vpzvdr7ba1jdnwqi6-python3.8-jupyter_client-6.1.12/bin/.jupyter-kernelspec-wrapped", line 6, in from jupyter_client.kernelspecapp import KernelSpecApp File "/nix/store/zkis5is358n0b4cq6878i0756rbvv6iz-python3.9-jupyter_client-7.0.2/lib/python3.9/site-packages/jupyter_client/init.py", line 6, in from .asynchronous import AsyncKernelClient # noqa File "/nix/store/zkis5is358n0b4cq6878i0756rbvv6iz-python3.9-jupyter_client-7.0.2/lib/python3.9/site-packages/jupyter_client/asynchronous/init.py", line 1, in from .client import AsyncKernelClient # noqa File "/nix/store/zkis5is358n0b4cq6878i0756rbvv6iz-python3.9-jupyter_client-7.0.2/lib/python3.9/site-packages/jupyter_client/asynchronous/client.py", line 6, in from jupyter_client.channels import HBChannel File "/nix/store/zkis5is358n0b4cq6878i0756rbvv6iz-python3.9-jupyter_client-7.0.2/lib/python3.9/site-packages/jupyter_client/channels.py", line 12, in import zmq.asyncio File "/nix/store/pvqmavsnzqp5m9shgkj69gqhrzjlfkgl-python3.9-pyzmq-22.2.1/lib/python3.9/site-packages/zmq/init.py", line 103, in from zmq import backend File "/nix/store/pvqmavsnzqp5m9shgkj69gqhrzjlfkgl-python3.9-pyzmq-22.2.1/lib/python3.9/site-packages/zmq/backend/init.py", line 32, in raise original_error from None File "/nix/store/pvqmavsnzqp5m9shgkj69gqhrzjlfkgl-python3.9-pyzmq-22.2.1/lib/python3.9/site-packages/zmq/backend/init.py", line 27, in _ns = select_backend(first) File "/nix/store/pvqmavsnzqp5m9shgkj69gqhrzjlfkgl-python3.9-pyzmq-22.2.1/lib/python3.9/site-packages/zmq/backend/select.py", line 32, in select_backend mod = import_module(name) File "/nix/store/66fbv9mmx1j4hrn9y06kcp73c3yb196r-python3-3.8.9/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "/nix/store/pvqmavsnzqp5m9shgkj69gqhrzjlfkgl-python3.9-pyzmq-22.2.1/lib/python3.9/site-packages/zmq/backend/cython/init.py", line 6, in from . import ( ImportError: cannot import name 'constants' from partially initialized module 'zmq.backend.cython' (most likely due to a circular import) (/nix/store/pvqmavsnzqp5m9shgkj69gqhrzjlfkgl-python3.9-pyzmq-22.2.1/lib/python3.9/site-packages/zmq/backend/cython/init.py)

Googling this error leads to github issues about mismatched python versions.

[Etjean]

I tried to run jupyter kernelspec list in the provided nix-shell, and everything worked fine. You can pin the exact revision of jupyterWith to use, like try the most recent one (since it worked for me), or one that made your shell.nix work before it started failing. For example:

let 
    jupyter = import (builtins.fetchGit {
        url = https://github.com/tweag/jupyterWith;
        rev = "9d9e85584e476aa1e461876f282e6bab261acd6f";
    }) {};

[choener]

I have stumbled upon the same problem. While jupyter kernelspec list shows the ipython kernel, once you run the notebook you end up with the above problem.

[garbas]

This will be fixed and documented with next version of jupyterWith (see main branch)