How do I connect to a remote kernel via the Jupyter Notebook Server API?

[rabernat]

I am trying to write a script interact with binder remotely. My approach is based on code written by @yuvipanda for hubtraf: https://github.com/yuvipanda/hubtraf/blob/master/hubtraf/user.py

Specifically, I am doing this, all using aiohttp:

• Use the binder API to build and start a binder
• When the binder is "ready", get the token and url to connect to the notebook server
• Use the notebook server API to start a kernel
• Connect to the kernel's websocket and make it execute code

For the last step, Yuvi wrote some fairly low-level code to send and receive messages via the websocket. But isn't this what jupyter_client does?

My question is: once I am connected to a Jupyter Notebook Server and can talk to its API, how do I plug in jupyter_client to talk to the kerneL? Is this possible? Does my question make sense?

The jupyter_client client API docs have a lot of methods I would like to use, but the docs seem to be missing some high-level intro about how the different parts work together.

[yuvipanda]

@rabernat I too wish jupyter_client could be used for that. Unfortunately, jupyter_client works when talking to kernels locally (via 0mq). However, we wanna talk to kernels remotely over websockets, and afaict there's no python client code that does that. There's a fair bit of JS code, but not Python.

I'd love for someone to do this, or at least re-use parts of jupyter_client.

[davidbrochart]

I'm wondering how this is related to https://github.com/jupyter/kernel_gateway. Maybe Binder could provide the ability to launch headless remote kernels, instead of having jupyter_client support kernel management through websockets?

[athewsey]

+1: I've been exploring automation solutions that connect to a JupyterServer through HTTPS REST+WebSocket APIs, and it feels painful to be re-implementing so much (in Python) when there's a reference right here... Perhaps it could make sense within this project's scope to support HTTP(S)+WS(S) as an alternative connection/transport layer?

[davidbrochart]

I guess we would need to deserialize/reserialize from/to ZMQ sockets to/from websockets, as it's done in Jupyter Server: https://github.com/jupyter-server/jupyter_server/blob/master/jupyter_server/base/zmqhandlers.py And yes I think it would be helpful.

[kevin-bates]

@athewsey - I'm not sure if this would satisfy your use case but if you fronted your kernels with a gateway server, you could essentially hit that server from a jupyter_client/nbclient-based application using a GatewayKernelManager/GatewayKernelClient as described here: https://github.com/jupyter/enterprise_gateway/issues/972#issuecomment-847262335

Pushing the zmqhandlers down closer to the kernel is something that would open the doors for some interesting applications for sure.

[davidbrochart]

Actually, the need to have a bridge between ZMQ sockets and websockets is only due to the fact that we pass through a server to access the kernel, but it doesn't have to be so. TCP ZQM sockets are accessible remotely, we just need to know their IP address and port number (basically, the connection file).

[yuvipanda]

github.com/quansight/jhub_client is what I use

[davidbrochart]

Thanks, I think it is https://github.com/Quansight/jhub-client.

[davidbrochart]

I have started https://github.com/davidbrochart/jpterm, that will allow to execute some code or a notebook through a Jupyter server.