Way to get kernel name and/or kernel spec from the running kernel?

[rcthomas]

I was wondering if there was a way, from within a notebook running a particular kernel, to find out either the name of the kernel or directly get the kernel-spec used to launch it? In particular I'm interested in inspecting the argv.

For context, we use a "helper script" pattern where we prefix argv with a kind of entrypoint script:

{
 "argv": [
  "{resource_dir}/kernel-helper.sh",
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "mykernel",
 "language": "python"
}

Where a user can specify the contents of an executable kernel-helper.sh like:

#!/bin/bash
module load python
module load other-thing
source activate whatever
export THIS_OR_THAT=1
exec "$@"

(Maybe there's something better for those kinds of customizations and we can retire this pattern?) The idea here is that it would be nice for the kernel running the notebook to be able to grab the kernel-spec and use the helper to run something else, like dask-whatever from the same environment with the same setup, independently (e.g. not as a child process of the kernel).

One thing I've seen so far is I can parse the name of the connection file to extract the ID and then query the API to ultimately get to it. Is there a nicer way to do this that I haven't found in searching the docs yet? I'm not 100% sure I am asking in the right place.

[kevin-bates]

Hi @rcthomas, You might want to take a look at a recently added feature to jupyter_client called Kernel Provisioning. This introduces a framework that essentially abstracts the Popen portion of kernel management, thereby allowing for custom kernel provisioners to provision the environment in which the kernel is running, yet still be managed by the Kernel Manager and hosting application.

Provisioners have access to the kernelspec, although conveying the kernelspec (or a portion thereof) to the kernel would still need to occur. This probably isn't a great leap from what you're already doing, but, with provisioners, you can probably embed much of what you do today in the provisioner and have exclusive access to the kernel process launch (however the provisioner chooses to accomplish that).

We also hope to introduce kernel parameterization at some point and that might be something to keep an eye out for.

[rcthomas]

@kevin-bates thank you, I had taken a look at the kernel provisioning docs you linked and thought it might help me either retire or at least streamline the kernel-helper pattern we've shared with users. I think we just need to move ahead and experiment with the provisioning mechanism a bit and see if we can figure out how to leverage it here, and follow-up with questions we come up with.

[rcthomas]

I have one question at this point, I think I see how the kernel provisioner can be used to get the kernel name to the notebook. Extending my earlier kernel-spec example a bit:

{
 "argv": [
  "{resource_dir}/kernel-helper.sh",
  "python",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "test99",
 "language": "python",
 "metadata": {
  "debugger": true
 },
 "env": {
  "KERNEL_NAME": "test99"
 }
}

Getting a kernel provisioner to inject the KERNEL_NAME environment variable would be enough to give the notebook a way to pull up the kernel-spec JSON later through a call through jupyter_client's kernel spec discovery feature. That would be very transparent to the user, they wouldn't have to add that redundant variable to their kernel-spec.

The question is: It seems useful to have this for any kernel, is there a reason not to just put such a variable setting in the base provisioner? Maybe there's a more preferred way to pass that information to the notebook than an environment variable?

I think that I do see how {resource_dir}/kernel-helper.sh could be prefixed onto argv by overriding pre_launch. The mechanism could be like, if there's such an entrypoint file located at the expected place then prefix it onto argv and run, otherwise just go with the argv in the kernel-spec. As long as the modified argv is re-constructible by the notebook that would make things a bit easier. The user would just need to provide the entrypoint script, but wouldn't need to touch the kernel-spec file.

[kevin-bates]

Hi @rcthomas.

It seems useful to have this for any kernel, is there a reason not to just put such a variable setting in the base provisioner?

I suppose there's some utility in conveying the kernel (specification) name to the kernel process, and using the environment to make the connection would certainly work across kernels and (possibly) provisioners and could be built into the framework if desired.

Your use case, assuming I understand it, appears to only pertain to locally running kernels since it seems to assume the kernel is going to make use of the kernel specification (kernel.json) file once it knows the spec name. As a result, this should only apply to the LocalProvisioner (or a subclass thereof) and not KernelProvisionerBase.

I think the bigger question is how is the code that acts on this value going to get into each kernel process? If you have a "wrapper" kind of mechanism that can interpret these values, then that might help, but otherwise you're looking at kernel modifications.

Also, one can have duplicate kernel specifications configured on a system but "partitioned" by runtime environments (python/conda virtual envs), so if the intention is for the kernel to read something from the filesystem, conveying a value that includes the absolute path is probably a better bet for success. And, again, assumes the kernel is local.

I would recommend a local provisioner subclass be implemented to achieve your desired functionality. Then, depending on how that goes and how applicable the solution is to the general mechanism, we could see if we want to make this core functionality in LocalProvisioner.

I hope that helps.

[rcthomas]

Thank you, I appreciate the guidance.