Add support for a C++ kernel

[kevin-bates]

C++ is a popular kernel that we currently do not support. This issue would consist of implementing a kernel launcher/wrapper that embeds the C++ kernel and sets up a listener a port on which enterprise gateway can communicate.

It also requires implementation of a kernelspec set of files to launch the kernel against the appropriate resource manager. Focus on kubernetes would be preferred.

This could be done using this C++ kernel.

[jashrathod]

Hi. I would like to work on this issue.

[lresende]

Let us know if you have any questions to get started. A quick look at our docs and at the latest #529 pr might help. I would recommend a vanilla C++ kernel as the first step, where we can distribute it in a kubernetes/docker env and then look at the possibility to also support YARN as a second step.

[jashrathod]

I do not have any experience with kernels, but would like to get into it. I went through the links provided but couldn't understand much. It would be great if I could get some help getting started with C++ kernels.

[kevin-bates]

Hi Jash - thank you for your interest. If you're new to Jupyter notebooks in general, this will be challenging, but nothing we can't overcome. Along the lines of Luciano's suggestion, we should start from known/working environments, then move forward. This issue is more about implementing a C++ kernel wrapper that sets up the C++ kernel for use in remote environments. As Luciano mentions, you'll want to focus on the containerized environments (I think we really meant to reference PR #528).

Let's do this using the Xeus-cling C++ kernel. The reason I mention that is because its primary contributor is also a Jupyter Steering committee member, so there's definite working knowledge relative to Jupyter.

Here are some possible steps.

1. Setup up a Jupyter notebook environment that uses the Xeus-cling kernel. Confirm that works.
2. You'll want to locate the kernelspec (kernel.json file) that is used to get an understanding of how the Jupyter framework goes about starting a C++ kernel. This command can be used to locate all available kernelspecs: jupyter kernelspec list
3. The primary work, is then to build a kernel wrapper that does the following:
   1. Creates a listener thread to listen on a port to which the Gateway will send interrupt and shutdown requests.
   2. Creates the 5 zero MQ ports and key information used for communication with the kernel.
   3. Launches the C++ kernel with the necessary 5 Zero MQ ports and key info.
   4. Sends the kernel port information back to the Gateway on the response address that the wrapper is launched with.
   5. Then just listens for requests form the Gateway - issuing interrupts as needed and terminating itself (and possibly the kernel instance) when shutdown is requested.

You'll want to model this wrapper after the ToreeLauncher or launch_ipykernel.py script.

Once the wrapper is ready, then you'll incorporate it, along with installation of the C++ kernel into kernel image. We should probably look to see if there are any C++ kernel images out there, but, if not, we could probably just extend elyra/kernel-py for starters.

I hope that helps. Just one step at a time. :smiley:

[lresende]

Hi Jash - thank you for your interest. If you're new to Jupyter notebooks in general, this will be challenging, but nothing we can't overcome. Along the lines of Luciano's suggestion, we should start from known/working environments, then move forward. This issue is more about implementing a C++ kernel wrapper that sets up the C++ kernel for use in remote environments. As Luciano mentions, you'll want to focus on the containerized environments (I think we really meant to reference PR #528).

I was mentioning the DASK PR, to have a reference to what artifacts need to be added/updated, but suggesting to focus on K8s instead of YARN for the first step.

Let's do this using the Xeus-cling C++ kernel. The reason I mention that is because its primary contributor is also a Jupyter Steering committee member, so there's definite working knowledge relative to Jupyter.

+1

[Icc-o]

Hello , I wonder if you have done this? I am looking for a c++ kernel.

[kevin-bates]

Hi @Icc-o - no integrating a C++ kernel so that it can be run within a resource-managed cluster via EG has not been completed. Are you interested in making that contribution? If so, we would be happy to help with the integration aspects.

If you're looking for a C++ kernel to use locally with Jupyter, I suggest the Xeus-cling C++ kernel we reference above. Otherwise, you'll need to look elsewhere for a C++ kernel that can run remotely - likely until this issue is resolved via a PR.

[Icc-o]

Thanks @kevin-bates . I'm interested in working on this . I'll try these steps. Maybe need your more help later.

Hi Jash - thank you for your interest. If you're new to Jupyter notebooks in general, this will be challenging, but nothing we can't overcome. Along the lines of Luciano's suggestion, we should start from known/working environments, then move forward. This issue is more about implementing a C++ kernel wrapper that sets up the C++ kernel for use in remote environments. As Luciano mentions, you'll want to focus on the containerized environments (I think we really meant to reference PR #528).

Let's do this using the Xeus-cling C++ kernel. The reason I mention that is because its primary contributor is also a Jupyter Steering committee member, so there's definite working knowledge relative to Jupyter.

Here are some possible steps.

1. Setup up a Jupyter notebook environment that uses the Xeus-cling kernel. Confirm that works.

2. You'll want to locate the kernelspec (kernel.json file) that is used to get an understanding of how the Jupyter framework goes about starting a C++ kernel. This command can be used to locate all available kernelspecs: jupyter kernelspec list

3. The primary work, is then to build a kernel wrapper that does the following:

   1. Creates a listener thread to listen on a port to which the Gateway will send interrupt and shutdown requests.
   2. Creates the 5 zero MQ ports and key information used for communication with the kernel.
   3. Launches the C++ kernel with the necessary 5 Zero MQ ports and key info.
   4. Sends the kernel port information back to the Gateway on the response address that the wrapper is launched with.
   5. Then just listens for requests form the Gateway - issuing interrupts as needed and terminating itself (and possibly the kernel instance) when shutdown is requested.

You'll want to model this wrapper after the ToreeLauncher or launch_ipykernel.py script.

Once the wrapper is ready, then you'll incorporate it, along with installation of the C++ kernel into kernel image. We should probably look to see if there are any C++ kernel images out there, but, if not, we could probably just extend elyra/kernel-py for starters.

I hope that helps. Just one step at a time. 😃

[kevin-bates]

Since a C++ kernel probably doesn't need to be launched within a Spark cluster (since Spark languages are Scala, Python, or R), I'm thinking we could use launch_ipykernel.py and convert it into a general "other kernel" launcher where it essentially launches a process (just like the other kernel's native launcher must do). The "launch_other.py" script would then manage the process and send it signals for interrupt, etc. To start the process would consist of building the connection file and handing that file to the target kernel (like it would do locally today). This launcher would then be general enough to support any kernel that is its own process.