Jupyter terminal using different sys.path from Jupyter notebook (running on same server, same conda env)

[whittredge]

I was having a common package import issue that was mostly resolved by several similar issues (https://github.com/jupyter/notebook/issues/2359, https://github.com/jupyter/notebook/issues/397, https://github.com/jupyter/notebook/issues/2898), but now I have a new issue that I don't think has been mentioned here. I am using conda on macOS 10.12.

If activate a conda env with source activate $envname, then run python from my local terminal, I can easily import the packages that I have installed in my envname.yml file. From this conda env I can start a jupyter notebook and run a Jupyter terminal. In the Jupyter terminal, sys.executable returns the same result as in my local terminal, and I can import all the same packages. HOWEVER, if I start a new Jupyter notebook from the same launcher, running side-by-side the functioning Jupyter terminal on the same server, sys.executable returns a DIFFERENT path (one that I thought I had deleted). Of course, this means I cannot import the package I hope to use in the notebook. Not only that, but the list of sys.path locations in the Jupyter terminal versus the Jupyter notebook are completely non-overlapping---see screenshots below.

It seems from the above screenshots that the Jupyter terminal is choosing to run python from a different conda env than the Jupyter notebook. To solve this problem I removed the wrong env that the notebook was using as sys.path with conda env remove. Now, there are no kernel errors when I start the Jupyter launcher or a Jupyter terminal, but starting a notebook returns a kernel error, as if it is expecting to work only with the path listed in the removed conda env.

[takluyver]

You've installed a kernelspec at some point which points to the other environment. Run jupyter kernelspec list to see where it is. If you delete it, it should fall back to running a kernel in the same Python that the notebook server is running in.

[whittredge]

Thanks for the speedy reply. I ran jupyter kernelspec list, then navigated to the dir listed for python3, then deleted the file kernel.json.

Unfortunately the Jupyter terminal and notebook are still in disagreement, as started from a new Jupyter launcher from the desired conda env:

[whittredge]

Update: I made a new dir, and created a new conda env. Now, the Jupyter terminal and notebook are in agreement that sys.executable is '/Users/.../anaconda/bin/python', which I think is wrong. The local terminal in the same conda env correctly identifies sys.executable as '/Users/.../anaconda/envs/new-env-name/bin/python'. Naturally I cannot import packages in the notebook that have been installed with conda install and confirmed present in the new-env-name.yml output.

[takluyver]

Have you installed Jupyter in the env you're talking about?

[whittredge]

With the conda env activated, which jupyter from my local terminal returned the expected path, /Users/.../anaconda/envs/correct-env/bin/jupyter. However, in a Jupyter terminal run from that env, which jupyter returned the root installation, so I'm not sure if this means I had the correct Jupyter installed in the env or not.

This issue was ultimately resolved by installing the package nb_conda (https://docs.anaconda.com/anaconda/user-guide/tasks/use-jupyter-notebook-extensions). Now, when I launch Jupyter from the conda env and click the "New" dropdown, I am presented with the notebook kernel choices "Python 2", and "Python [correct-env]". The Jupyter terminal also returns the expected path for which jupyter.

Thanks for your help!

[knaftchi]

I had the same issue, and nb_conda fixed it for me. Essentially, as whittredge explained, Jupyter was looking into a path different than my virtual environment. After installing nb_conda, I was able to start Jupyter notebook and then select the specific kernel I wanted it to access (in my case, the specific virtual env I had created). It works now. Thanks for the tip, whittredge.

[apano]

Hi, I am still experiencing this issue. I installed nb_conda again and what I see is that pip is using the root conda installation : /opt/conda/bin/pip sys.executable shows: '/home/jovyan/conda-envs/python3.7.1/bin/python' which jupyter shows: /opt/conda/bin/jupyter

The issues I encounter are only in the notebook and not in the terminal. The terminal shows correct paths.

[themataleao]

I was able to solve the problem by installing jupyter notebook within my new environment. First acitvate your environment: activate py36 (I called my environment py36) Seconnd install jupyter notebook within that environment: pip install jupyter notebook Then launch our jupyter notebook jupyter notebook

[y-x-c]

delete kernel.json did the trick for me.

[rsatijaUT]

Thanks @swissrapperswil. I have both Canopy and Anaconda, and due to an update, the default switched to Canopy. For this reason, the conda packages, while running smoothly using bash would be unrecognizable in jupyter notebooks. Simply installing the jupyter notebook in the default (canopy) environment solved it.

[engmubarak48]

Have you installed Jupyter in the env you're talking about?

Actually, such a problem comes when you have conda environment and you install jupyter via pip instead of conda. It happened to me, I had jupyter running but couldn't import libraries.

[rraadd88]

Changing the executable path of jupyter (in the kernel.json) worked in my case. For details see https://stackoverflow.com/a/57274684/3521099 .

[theccalderon]

With the conda env activated, which jupyter from my local terminal returned the expected path, /Users/.../anaconda/envs/correct-env/bin/jupyter. However, in a Jupyter terminal run from that env, which jupyter returned the root installation, so I'm not sure if this means I had the correct Jupyter installed in the env or not.

This issue was ultimately resolved by installing the package nb_conda (https://docs.anaconda.com/anaconda/user-guide/tasks/use-jupyter-notebook-extensions). Now, when I launch Jupyter from the conda env and click the "New" dropdown, I am presented with the notebook kernel choices "Python 2", and "Python [correct-env]". The Jupyter terminal also returns the expected path for which jupyter.

Thanks for your help!

This saved my life! Thank you so much!

[Kimonili]

I have the same issue but, at the same time, kind of different.

sys.executable gives the same result in both terminal and notebook but sys.path is different. I updated the PYTHONPATH to include some modules I coded myself, but while they are successfully imported when I run python from terminal without any exceptions when I do the same from the notebook I get No module named < name of module > error.

Any suggestions?

[connorferster]

I am having a similar issue except the terminal gives sys.executable and sys.path as my conda environment (correct) but from the notebook they both point to my base environment. It's strange because I have created many conda environments and with their own jupyter installation before and I have never had an issue with this. My guess it has something to do with the newest jupyter lab version that I installed into a new conda environment but I cannot confirm this yet.

I find it so puzzling that the notebook and the terminal launched from within jupyter point to different environments! So strange!

[kevin-bates]

The notebook information reflects the kernel's environment, while the terminal information reflects that of the notebook server. I suspect your kernel configuration is launching a python relative to your base environment.

Run jupyter kernelspec list to identify the location of your kernel.json file. Take a look at the first argument in the argv stanza, which should identify the python executable in which the kernel will reside. If it's fully qualified, you might try making it relative (i.e., just python or python3, or even sys.executable) and let the default resolution behavior determine which python should be invoked.

This is probably similar to https://github.com/jupyter/notebook/issues/3311#issuecomment-516459025 (now that I see that response above), although it appears that that solution still uses an absolute path - which should be fine as well. If you take the relative approach I suggest above, the resolution is probably more a function of your PATH setting and should always equate to the same python environment in which the notebook server resides.

[menosgeze]

For people using the strategy of installing jupyter notebook within the environment and then running it, it may be worth to check which jupyter you are activating by default. Run first which -a jupyter and then select full path to the one in your environment, then your list will show something like Python 3(ipykernel) in the list, but in my case it does not show the Python 3 install by default in my mac.

[ks905383]

Would like to second @engmubarak48 ; this problem was fixed for me when I changed the environment.yaml file I was using to generate the environment to include jupyter and jupyterlab. Otherwise it was using the base environment's jupyter (even when running it after deactivating the base environment, which felt odd), with the incorrect paths.

[Step-by-step1]

Had similar issue, only I was using pipenv as virtual environment. After running pipenv install jupyter within virtual environment, a ran ipython kernel install --name "local-venv" --user. After that, when I ran jupyter notebook in that virtual environment, I was able to select new kernel where all respective packages were available. https://queirozf.com/entries/jupyter-kernels-how-to-add-change-remove

[ShidongPAN]

With the conda env activated, which jupyter from my local terminal returned the expected path, /Users/.../anaconda/envs/correct-env/bin/jupyter. However, in a Jupyter terminal run from that env, which jupyter returned the root installation, so I'm not sure if this means I had the correct Jupyter installed in the env or not.

This issue was ultimately resolved by installing the package nb_conda (https://docs.anaconda.com/anaconda/user-guide/tasks/use-jupyter-notebook-extensions). Now, when I launch Jupyter from the conda env and click the "New" dropdown, I am presented with the notebook kernel choices "Python 2", and "Python [correct-env]". The Jupyter terminal also returns the expected path for which jupyter.

Thanks for your help!

Thanks, I encountered the same problem, and resolved by installing `nb_conda''. :)