Julia: Add variable sharing

[ShalokShalom]

Hi there :)

I would be interested to add a new language.

From the looks of it, it seems pretty doable, and I am still confused, as to why some of the languages are implemented in this repo, and some, like Python seem to be difficult to locate at all, and Javascript has its own location? altogether.

The most insightful information I could find, was the FAQ.

Still, I welcome a little more details, as to how I could add Julia. It runs mostly in a JIT, that evaluates LLVM code, if that is any helpful.

[jonsequitur]

Julia should already work, though we're obviously missing documentation for it.

.NET Interactive's Python and R support are via the Jupyter protocol, so other languages that have dedicated Jupyter kernels can work via those kernels.

https://github.com/dotnet/interactive/blob/main/docs/jupyter-in-polyglot-notebooks.md

If you try it, we'd be curious to know if you run into issues.

[ShalokShalom]

Oh wow.

I am way too sleepy now, but I am going to look into this tomorrow. Wonderful day 🌞

[shibbas]

Yes, Polyglot kernel can connect to IJulia using the Jupyter support. However, Jupyter protocol does not support variable sharing so that will need to be added custom. Here are the instructions on how you can add that for the Jupyter sub-kernel: https://github.com/dotnet/interactive/blob/main/docs/adding-jupyter-kernels.md

IJulia does support Jupyter comms message so it can be added.

[ShalokShalom]

Thanks a lot. That is a very helpful guidance. I will add a PR to the README, that links to that documentation.

[jonsequitur]

I'll make some docs updates and leave this issue open as the updated title reflects something that hasn't been done. There isn't currently support for Julia variable sharing. We'd happily help with a PR if anyone is feeling motivated to contribute.

[johnW-ret]

Is it possible that custom LanguageHandlers could be exposed via some API or config? Or maybe we could get documentation on building from source and dogfooding? I got OCaml working via Jupyter and I'd also like to add variable sharing, but as Jupyter support is unofficial for OCaml and we already have F#, I don't expect there will be much support for adding official support for the language.

[jonsequitur]

It's a good suggestion to make this extensible. @shibbas might have some thoughts on that.

Documentation for developers is here: https://github.com/dotnet/interactive/blob/main/DEVELOPER-GUIDE.md

[shibbas]

@johnW-ret totally agree with you on passing this in for non-default supported languages. This was very much the reason for providing the command line option for initializing the kernel with custom scripts using --init-script where you can pass in the script to set up your custom comm target before checking for variable sharing.

Currently we only check for variable sharing if the script is under LanguageHandlers and not always because some of the Jupyter kernels don't seem to correctly respond to comm_open requests when the handler isn't there (e.g. Julia doesn't send a comm_close message I believe) and this could cause the system to be in a weird/hung state. But an an option to force check variable sharing via command line options could be added. Something like:

#!connect jupyter --kernel-name <kernel-name> --kernel-spec <kernel-spec-name> --init-script <your-variable-sharing-script> --share-variables

Alternatively, instead of force check, perhaps a timeout might be better to ensure that the kernel will always be in a good state even if the kernel doesn't respond as expected to a comm message. Here: https://github.com/dotnet/interactive/blob/5c255464dc9af2a12409d0609525c981b58dc63a/src/Microsoft.DotNet.Interactive.Jupyter/CommandEvents/CommCommandEventChannelConfiguration.cs#L75C1-L86C14

Either way, a small change would be needed in here to bypass initializing the comm targets only for LanguageHandler scripts: https://github.com/dotnet/interactive/blob/5c255464dc9af2a12409d0609525c981b58dc63a/src/Microsoft.DotNet.Interactive.Jupyter/CommandEvents/CommCommandEventChannelConfiguration.cs#L51C5-L67C6

[johnW-ret]

@shibbas This is awesome. My next steps will be making those changes on my fork and writing a OCaml coe_comm_handler.

One question though: I see that I can load --from-file to the value kernel and @ reference it in --init-script. Is this a good workflow we should recommend for variable sharing with custom comm targets? I don't imagine it will need to be done often. And if so, can we get a bit of markdown somewhere in the docs for it?