Need support for debugging files

[DinoV]

In addition to debugging a running kernel, presumably we want to support debugging files as well. This comes in two variants. There's the "step-in" experience where a user goes from cell code to a library and there's the "launch and debug a file" in a more traditional IDE experience.

The step-in experience really just speaks to the UI - users should be able to set breakpoints in files, step-in should work and open/focus the file, etc...

The "launch and debug a file" scenario is more nuanced. There's several options on how to handle what the debug communication is here and they're closely related to decisions in issue #6.

1. Use the existing VS Code protocol entirely as is for files and iopub/shell for kernels. Jupyter then needs to understand both it's envelope for messages as well as the standard VS Code envelope. For full support languages need to support producing both envelopes. Existing VS code languages would be debuggable in Jupyter as plain old files.
2. Use the Jupyter ZMQ shell + io_pub channels for debugging even "file run" operations. Debuggees would be started with both ports specified. Existing VS code languages wouldn't work. Kernels would only need to support the envelope format they're currently used to supporting.
3. Support the ZMLQ shell + io_pub format for file debugging, but provide an adapter in-box which supports the VS Code format. The notebook side of things would be unmodified and consume shell/iopub. The Python side of Jupyter would contain the adapter. Configuration would specify the communication channel used and the adapter would be used when configured. Language implementers could then choose to just speak existing shell/iopub ZMQ or to use a combo. Existing VS code languages would all be debuggable in Jupyter as plain old files.
4. Use the VS Code protocol for debugging as-is and don't move it into the shell/iopub channels. Give the kernel a new port for it to connect back to, and use the VS Code IPC format. This would mean kernels would now be dealing with two different forms of IPC - ZMQ + the more HTTP style VS code format. Both kernel debugging and file debugging would use the same mechanism. Existing VS code languages would be debuggable in Jupyter as plain old files.
5. Move Jupyter to a VS Code like kernel connection. Unify to a single port, extend the message API with support for kernel request/replies/events. Provide an adapter which maps iopub/shell based kernels into this new message spec. Update all of Jupyter to consume the new format. This would likely break kernel + notebook extensions which are sending extended messages over the existing channels as they may not be successfully mapped. It's also a significant architectural change which will touch everything.
6. Add a new debugger channel This keeps the existing Jupyter message envelope, but runs the messaging over a new channel. An adapter as defined in option 3 could still exist. This would accommodate issue #9

It seems to me that option 3 makes the most sense... It gives debugging of all VS Code languages immediately, let's kernel authors choose what's best for them, and keeps consistency of the Jupyter kernel channels.

Regardless there needs to be some updated registration for how to "run files" potentially in a kernel spec or elsewhere in Jupyter's configuration system.

[fperez]

One thing I don't fully understand in option 3: would the proposed adapter live in-kernel? If so, you mention it being written in the "python side of Jupyter", but for many kernels, there's no python process at all, they implement the ZMQ transport and JSON encoding themselves.

I suspect I'm just misunderstanding your proposal a little...

ps - I minimally edited your original message to bold the summary statements, I think it makes it more readable. I hope you won't mind :)

[DinoV]

It would live in the notebook package. Somewhere in configuration it would specify which protocol to use, and then notebook would spin up the adapter and launch and send JupyterLab only the adapted ZMQ based protocol.

Improved formatting is always welcome :)

[fperez]

Ah, ok. It seems this would entail adding to the kernel spec information about whether the kernel supports the VSCode format, so the Jupyter server can then adapt those packets when it opens the connections to the kernel and present them via the existing ports to downstream clients. Does that sound right?

[DinoV]

Yes, although I had originally been thinking that this would apply only to the "run file" case, so whether that's done by registering a kernel or some other sort of registration would be TBD.

Mainly I see the adapter as getting a breadth of languages which are already supporting the VS Code protocol for running files. But given that no kernels support debugging today there's no specific advantage there. I suppose if there was a language which supported VS code debugging already, and they wanted to add Jupyter kernel support, it could potentially reduce the burden on them. But that's also something that could just be added later too.