Document how to create a shared conda environment

[benbovy]

Update by Erik 2021-10-25

A sensible pattern to do this is by doing either...

$ sudo -E conda install -c conda-forge nb_conda_kernels
$ sudo -E conda create -c conda-forge -n <conda-env-name> python ipykernel
$ sudo tljh-config reload hub

Or following #697, by doing...

$ sudo -E mamba install nb_conda_kernels
$ sudo -E mamba create -n <conda-env-name> python ipykernel <other packages>
$ sudo tljh-config reload hub

Original issue

I'd like to to deploy jupyterhub on a bare-metal server using TLJH (BTW, thanks for providing this well-documented tool, this looks like it will make things a lot easier for me!).

In our team we're used to maintain separate conda environments for each of our projects, and use nb_conda_kernels to easily switch between kernels within the notebook or lab interface. This is much easier to maintain than one single, big conda environment.

As I understand, TLJH only provides one shared conda environment. How easy/hard would it be to add support in TLJH for multiple conda environments shared between all users + nb_conda_kernels?

[benbovy]

Maybe this could be just as easy as that?

$ sudo -E conda install -c conda-forge nb_conda_kernels
$ sudo -E conda create -n project_name python numpy scipy
$ sudo tljh-config reload hub

[benbovy]

Mmmh maybe not as easy, since JupyterHub is run from a python3 virtual environment and there is no pip package for nb_conda_kernels.

[willirath]

And nb_conda_kernels depends on conda.

[willirath]

But you can install the ipykernel package in the new env and then do sth along the lines of

python -m ipykernel install --name ... --display-name ...

from the new env.

[benbovy]

Mmmh maybe not as easy, since JupyterHub is run from a python3 virtual environment and there is no pip package for nb_conda_kernels.

Or maybe I'm wrong here? The user (conda) environment is the one used to launch the notebook interface for all users, so installing nb_conda_kernels there could work after all?

I'll give it a try...

[benbovy]

So I tested it on a fresh tljh deployment and it worked well! I did no more than

$ sudo -E conda install -c conda-forge nb_conda_kernels
$ sudo -E conda create -n myproject python numpy scipy ipykernel -c conda-forge
$ sudo tljh-config reload hub

It would be great to have a custom hook to create new conda environments! That would be very useful for creating plugins that provide multiple environments!

[benbovy]

nb_conda_kernels also works nicely with user created conda envs (a user just need to create a new environment - with a kernel installed - from the terminal and then restart the notebook/lab server to see the kernel in the list). As user installed packages or envs has been discussed in other issues (e.g., #428), I'll try to see if I can put together custom hooks + doc updates in a new PR.

[benjaminjack]

Any further documentation on this? We are facing the same issue :)

[jetilton]

$ sudo -E conda install -c conda-forge nb_conda_kernels
$ sudo -E conda create -n myproject python numpy scipy ipykernel -c conda-forge
$ sudo tljh-config reload hub

@benbovy The above did not work for me. Did you do this in the tljh termina? I did and restarted my notebook I still only see Python 3 as my only option. Has there been any follow up on documentation?

Thanks

EDIT

Nevermind. I did the above, then nothing seemed to work, went to control panel and stopped/started my server and everything works as expected!

[willirath]

I have a few TLJH instances that did exactly this successfully. I'm not sure if it's also necessary to restart the user server, to see the new kernel, though. A page reload was definitely needed.

[benbovy]

I guess you need to restart your server once when you install nb_conda_kernels in the TLJH's user environment, but yes afterwards when you create new conda environments a simple page reload is enough to see them appear in the kernel list.

[mmccarthy404]

Would this still be the ideal way to deal with this in 2021?

[willirath]

Yes, I'm still using this in production a lot.

[meeseeksmachine]

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/tljh-packages-installed-for-user-instead-of-environment/11925/1

[vsisl]

I successfully added a conda environment based on @benbovy instructions (Thank you!)

Could you please advise me how to make this new environment the default kernel for all users and notebooks? 🙂

[willirath]

Never did myself, but this issue could have hints: https://github.com/Anaconda-Platform/nb_conda_kernels/issues/162

[vsisl]

Never did myself, but this issue could have hints: Anaconda-Platform/nb_conda_kernels#162

Thanks for a prompt reply @willirath ! I'll try to follow that guide.

But before I start, I want to ask about a couple of things. It is not clear to me what is the difference (if there's any) between the jupyter notebook config file, which can be generated by executing

$ jupyter notebook --generate-config

and the tljh config file, which can be viewed using

$ sudo tljh-config show

Are they the same thing?

What is the meaning of c. in c.MultiKernelManager.default_kernel_name?

Based on my gut feeling, I would try to adopt it for tljh in this way:

$ sudo tljh-config set MultiKernelManager.default_kernel_name my_conda_env

EDIT: Neither of

$ sudo tljh-config set MultiKernelManager.default_kernel_name my_conda_env
$ sudo tljh-config reload

and

$ sudo tljh-config set c.MultiKernelManager.default_kernel_name my_conda_env
$ sudo tljh-config reload

seem to work.

[jakub-kaminiarz]

@benbovy I've used your suggestions, managed to create conda environment. the env is visible for different users using conda env list however it is not visible from KERNEL tab. Any suggestions ?

[philipp-c]

I successfully added a conda environment based on @benbovy instructions (Thank you!)

Could you please advise me how to make this new environment the default kernel for all users and notebooks? 🙂

+1 for this question. @vsisl : did you manage to change the default kernel?

I tried the same thing, i.e. sudo tljh-config set MultiKernelManager.default_kernel_name my_conda_env as well as MappingKernelManager.

I also tried the actual name of the conda-env (as given with the -n flag when creating it) and the long name which I get from python -m nb_conda_kernels list

Unfortunately, no success so far.

[vsisl]

Hi @philipp-c

No, unfortunately I never figured out how to change the default kernel.

[jeroenwinkelhorst]

hi, I am also struggeling with a way to change the default kernel in TLJH. It looks like I solved it: I created a jupyter notebook config file with the following content (jupyter_notebook_config.py): c = get_config() c.MultiKernelManager.default_kernel_name = 'NAME_KERNELSPEC'

and placed this file in the folder /opt/tljh/user/etc/jupyter/ Now the defined kernelspec is the default in jupyter notebooks for all users