Swap ob-ipython out for emacs-jupyter?

[dangirsh]

I learned about emacs-jupyter yesterday. My main goal of this issue is to ensure the Scimax maintainers are aware of the emacs-jupyter effort and it's potential to level-up the Scimax <-> Jupyter interface.

Here are some interesting things about emacs-jupyter that stand out to me.:

• Direct zmq interface should provide snappier interactions, especially noticable for completions. Completion lag is one of the biggest gripes I have with ob-ipython.
• very active development + detailed documentation + well-organized code
• experimental client supports interactive widgets via an external browser
• easily associate org blocks and buffers to specific kernels (including remote ones).

Full disclosure, I haven't tried emacs-jupyter out yet. However, I've used ob-ipython with Scimax enhancements extensively (thanks!).

[jkitchin]

This looks great. I started an emacs zeromq lib myself over a year ago to move in this direction, but only got to a proof of concept stage. Most of the scimax enhancements could probably be used with this one day. Probably this will have to be an optional way to run ipython though, I can't imagine how windows users will build the requirements. everything else looks really promising though!

[dangom]

I just tried emacs-jupyter. It's truly impressive how responsive and robust it is. The REPL also feels like a work of art after seeing how well it behaves... and it supports inline plotting! Things just work as expected.

Now what's really cool is that much of scimax extensions to ob-ipython already work with emacs-jupyter, i.e., most of the notebook functionality (merging, splitting, etc). Other extensions, such as the completion and inspection functionalities are provided by emacs-jupyter, so there'd be no need to reimplement them. For example, typing from os.path import e on a src block and pressing TAB immediately opens a helm completion-at-point buffer with the expected completion candidates.

Only issues I found is that the REPL keybindings are different from the ones I've gotten used to. Also missed better header default args, and the clean separation and filtering of mime types that scimax provides, but that'd be a breeze to reimplement.

And apparently support for Windows is not impossible: https://github.com/dzop/emacs-zmq/issues/1 .

Overall a much better experience and extremely promising. Also for standard Python development the emacs-jupyter REPL offers a better experience than the comint based inferior python mode.

[jkitchin]

That sounds pretty promising. It does seem like we might need to keep the current solution around for windows. Maybe the newer windows that seem to have some linux shell possibility would make it feasible one day.

[fleimgruber]

We are making progress on the Windows side, see https://github.com/dzop/emacs-zmq/issues/3.

[nnicandro]

Hello,

Apologies for not responding to this post sooner as it is in reference to the project I'm the maintainer of.

@jkitchin I'd be happy to fix/improve emacs-jupyter to make it more feasible for you to potentially make the switch over to that project. What about the current Windows support do you find needs improvement? I know there is https://github.com/nnicandro/emacs-jupyter/issues/214, is that still an issue for you? Is there anything else besides Windows support you think could be improved?

Also I'd love to hear opinions on the direction emacs-jupyter should take moving forward. I've been working on addressing https://github.com/nnicandro/emacs-jupyter/issues/176 which may be of interest since it will make emacs-zmq an optional dependency. The related work is in emacs-jupyter/next. I plan to make a proper issue over at emacs-jupyter giving an overview of the changes and requesting some users to test. The work actually touches on more than making emacs-zmq optional. It updates the whole communication backend, from the way kernel messages get received by ZMQ sockets/Websockets and processed internally by the code base to the way those messages are handled by client code. It also introduces the publisher/subscriber pattern which I'm hoping will make it easier to add new features and fix bugs moving forward.

I plan on getting those changes merged in the short term. As for more medium to long term, I plan on addressing the issues that have built up and figuring out ways to get more contributors. I'm also looking for some next big feature to add to the project. Maybe supporting debug messages? Anyway, ideas would be appreciated.

[jkitchin]

I don't currently have a windows machine for testing, so I am not sure if there are still limitations there.

An option to not use zmq would be great if compiling is an issue for some. I noticed that Julia sometimes crashes emacs when I use it with emacs-jupyter, I assume that is because of the zmq code for some reason. I haven't found a recipe to reproduce the crash yet. I don't use it that often.

I have been using emacs-jupyter myself for a while. You can see what I am doing with it here https://www.youtube.com/watch?v=RD0o2pkJBaI. It is still a WIP, and some things there are differences in preferences.

I don't have any great ideas for new big features. It makes sense to be feature complete with the kernel, so debug messages could be great. I am still trying to figure out how the code works; I usually hit a wall around the generic functions and have trouble debugging and tracing the code there.

[nnicandro]

I'll have a look at updating my Julia installation and see if I run into any problems.

I watched your video. Thanks for promoting my project! It gives me a lot of points on how I can improve the Org integration. It seems a lot of people (based on the emacs-jupyter issue page) want to have more support for Org header arguments like :file, :wrap, :results raw and the like. From you video, it also looks like calling named Jupyter source blocks via other source blocks doesn't work properly. These are issues that I definately plan on addressing in the near future.

Could you give an example about where you have trouble tracing the code? Are you meaning when you instrument the function, you sometimes aren't dropped into the debugger? If so, I think that's because the function that does all the message handling is called by a process filter function.

[jkitchin]

A prototypical example of difficulty tracing the code is:

Say I open ob-jupyter.el, and am looking at the code for org-babel-jupyter--execute. The very first thing that is called is jupyter-send-execute-request, with I guess a lexically scoped jupyter-current-client. If I type M-. on jupyter-send-execute-request, it jumps to a line in jupyter-client.el that seemingly has nothing to do with that function (it is a line that starts with (define--jupyter-client-sender...). This is the generic function wall I seem to run into where it isn't easy to follow what is happening.

Another example is I can't seem to use edebug on one of the functions like ``org-babel-jupyter--execute` because it needs a local buffer variable that isn't available in the debug session.

[nnicandro]

I'll look into making the project more debuggable. Thanks for making these points. It's good to get some feedback on how the code can be structured to make it easier for other people to hack on it.

In the next branch that I'll be merging I got rid of the define--jupyter-client-sender macro so they won't be an issue.

I don't see any local buffer variable in org-babel-jupyter--execute, but I did fix an issue that sounds similar to what you are describing in org-babel-execute:jupyter https://github.com/nnicandro/emacs-jupyter/commit/0dec83f7e4a125c73430a45c2fb2119dcc31bd28.

If you find other problems with the code, feel free to open up an issue at emacs-jupyter. Especially if it's about something you think would make it easier for people to contribute to the project (like making the code easier to debug).

[jkitchin]

If I instrument org-babel-execute:jupyter, and step through it, it fails at something near (thread-last (buffer-local-value 'jupyter-current-client))

Wrong number of arguments: buffer-local-value, 1

is that the bug you are referring to? It works fine when the function is not instrumented.

[nnicandro]

Yes that's the one.

[jkitchin]

I can confirm that issue has gone away with the newest version on MELPA, thanks!