Usage of metadata for kernels in conda

[ellisonbg]

I am working with a team of interns to improve the experience of starting kernels in situations where kernels are managed in conda. Right now, conda is putting a very long string into the display_name that has the metadata related to the kernel (conda path, env name, etc.). We would like to improve the UI/UX for clients that are working with this information, such as in the JupyterLab launcher, but also things like nteract and classic. We are wondering if we should work with the conda folks (who would be the right folks?) to create a conda namespace in the kernelspec metadata that include structured information can be used by different UIs. Should this be more general than conda?

Ping @rgbkrk @i-am-am @takashimokobe @zuoyuanh @tgeorgeux @markelley @jennalandy

[ellisonbg]

ping @pzwang - who should we work with on the conda side?

Here is the JupyterLab issue related to this: https://github.com/jupyterlab/jupyterlab/issues/3795

[ellisonbg]

This is also relevant:

https://jupyter-client.readthedocs.io/en/latest/kernel_providers.html

[dhirschfeld]

I think this more accurately refers to nb_conda_kernels - conda itself doesn't have anything to do with jupyter kernels.

I use conda but prefer manually managing my kernels so don't have nb_conda_kernels installed.

[rgbkrk]

I'd love to have a conda namespace in the kernelspec metadata. I don't want to see this discussion be something broader and more generic, instead opting for pragmatic adoption of a nice little spec for conda.

[ellisonbg]

Thanks Kyle, that is helpful input!

Sent from my iPhone

On Oct 23, 2018, at 8:22 PM, Kyle Kelley [email protected] wrote:

I'd love to have a conda namespace in the kernelspec metadata. I don't want to see this discussion be something broader and more generic, instead opting for pragmatic adoption of a nice little spec for conda.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[msarahan]

I'm not familiar with nb_conda_kernels, but I'm happy to help implement any necessary changes on the conda side.

[mcg1969]

Hey folks, I've been working on the nb_conda_kernels project. Perhaps someone can file an issue over there with some details about what we should do?

Note that I've been experimenting with the kernel provider mechanism. My understanding is that it's not 100% stable but so far I've got something that will work.

[ellisonbg]

Hi @msarahan and @mcg1969 thanks for joining in! It isn't clear here where the right place for the changes is. Can we start to discuss it here and then open issues on other repos as needed.

A couple of question to get things started:

• how does nb_conda_kernels set the kernel name (what information is included)?
• If we create a new conda namespace in the kernelspec metadata, what attributes should be included? I am hoping the answer to the first question will guide this.
• Should kernels installed with both nb_conda_kernels and conda itself include this metadata in the kernelspec?

[mcg1969]

how does nb_conda_kernels set the kernel name (what information is included)?

By default, the format is '{0} [conda env:{1}]'.format(language, env_name), but it can be configured with the name_format option.

If we create a new conda namespace in the kernelspec metadata, what attributes should be included? I am hoping the answer to the first question will guide this.

For the most part, a kernel is uniquely defined by the environment name, defined by the basename of the environment directory, and the kernel name. The one wrinkle is that it's possible for environments to live outside of the root conda directory tree, which raises the potential for naming conflicts. But those can be readily handled.

That said, I'd like the flexibility to let nb_conda_kernels itself tell any sort of kernel manager what the "name" of the environment is. That gives us flexibility in cases where, for instance, an environment lives within some sort of larger project structure, so we might want to tell Jupyter the environment ID is some amalgam of the project name and environment name.

Should kernels installed with both nb_conda_kernels and conda itself include this metadata in the kernelspec?

I'm not sure I understand this question—nb_conda_kernels should not be installing kernels, at least not anymore. Rather, it should scan conda environments for kernels installed by conda. The latest version of nb_conda_kernels preserves as much of the native metadata as possible, modifying just the execution command so that it activates the environment properly.

[i-am-am]

After some discussion with @tgeorgeux and @takashimokobe, it seems like it was a mistake on our end to think that nb_conda is causing the issue we're having with long names.

We're planning to solve this with a limit on the length of display names, and go from there.

[ellisonbg]

@mcg1969 I looked more about how nb_conda_kernels is handling specs and have a proposal based on the logic that is here:

https://github.com/Anaconda-Platform/nb_conda_kernels/blob/master/nb_conda_kernels/manager.py#L131

The idea here is that nb_conda_kernels reads all the specs and then mutates them so they work and have different names. My proposal is this:

• Don't mutate the name of the kernels - let's kernels declare that themselves and leave it alone.
• Add conda specific metadata to the returned kernelspecs that frontends can use in a strucutured manner.
• Modify frontends to use the simplifed names + metadata in useful ways.

Are you open to this? Would take some coordination of the different projects, but I can help with that.