Tracking Issue: Interactive Computing with Jupyter

[rgbkrk]

Check for existing issues

• [X] Completed

Problem

As evidenced in https://github.com/zed-industries/zed/issues/5273, Jupyter Notebook users would love to edit notebooks. There's a need for seamless integration that supports the variety of media types in Jupyter notebooks and the interactive runtimes underneath.

Proposed Plan

The fastest path where we'll see progress along the way will be to:

• Create a kernel/runtime manager for Zed
• Enable execution in the editor, a la Hydrogen
• Create a notebook viewer
• Allow execution in notebooks

In parallel, we can work on new GPUI features to support rendering custom media types that Jupyter users expect.

Steps

Runtime support

To bring Jupyter kernels (aka runtimes aka REPLs) to Rust, the https://github.com/runtimed/runtimed project has been started.

• [ ] Discover python and/or jupyter kernel environments
• [ ] Launch kernels/runtimes
• [ ] Track kernels/runtimes in their own pane (?)

Notebook File support

• [x] New filetype support (https://github.com/zed-industries/zed/pull/9353, https://github.com/zed-industries/zed/pull/9425)
• [ ] Load Jupyter notebooks .ipynb into a new viewer

Rich Media -> GPUI needs

Some of the needs for notebook and/or in-editor execution will require GPUI work. Some will be able to reuse existing components as is (div, img, markdown preview).

These are the likely mediatypes to support:

• [x] text/plain -> div()
• [x] image/svg+xml, image/png, image/jpeg, image/gif
• [ ] application/vnd.dataresource+json - develop a custom table component to elegantly display tabular data within notebooks without needing text/html media type
• [x] text/markdown - presumed able to support based on markdown preview
• [ ] text/html - no support, yet highly needed
• [ ] text/latex - For rendering mathematical expressions.
• [x] application/json - likely to be made with a custom component or a read-only buffer
• [ ] application/geo+json
• [ ] VegaLite, Vega - unknown ability to support
• [ ] application/javascript - no support
• [ ] application/vnd.jupyter.widget-view+json, application/vnd.jupyter.widget-state+json - no support

Rich Media rendering

Same list as above, with actual implementation

• [ ] text/plain
• [ ] text/html
• [ ] text/markdown
• [ ] text/latex
• [ ] application/javascript
• [ ] application/json
• [ ] Image formats (image/svg+xml, image/png, image/jpeg, image/gif)
• [ ] application/vnd.jupyter.widget-view+json, application/vnd.jupyter.widget-state+json - no support
• [ ] application/geo+json
• [ ] VegaLite, Vega
• [ ] application/vnd.dataresource+json

[jianghoy]

Regarding text/latex, I believe today's best option is: MathJax(the default Tex implementation in webstack) -> Rust binding of MathJax running on node(the biggest unknown, talk about it below) -> SVG (which is supported by GPUI). Given Zed has a very low level UI support it should be reasonable to implement a mix of text, md, and SVG layout.

MathJax binding

Today there's a rust crate called MathJax, however, the level of support is questionable. And I feel using a headless chrome as nodejs backend for Mathjax is a big yellow flag.
There's also a KaTex rust binding, which uses quickjs(a small and fast and almost feature complete js engine implementation by Fabrice Bellard) or wasm js as node backend. But I don't think KaTex supports outputting to SVG (correct me if I'm wrong).
Hence, the best option forward I believe would be:

1. either raise a PR to the current MathJax rust repo and add quickjs as a backend; or create a new crate that does it;
2. also, it would be wise to setup CICD that automatically updates MathJax version in the rust binding; one concern I have over these 3rd party wrapper is that, when upstream releases a patch, downstream may not be responsive even though the change will be minimal. And CICD could help that.

Once these are sorted out, one can start verifying the work by adding Tex parsing into markdown similar to Jupyter and see if it renders correctly. This verification may or may not ship.

[vultix]

What’s the plan for supporting html rendering? This is an important feature for Jupyter, and I can think of a few other use cases for wanting an embedded browser / web renderer in Zed

[rgbkrk]

My first pass is going to involve using table schema instead of HTML for the most important use case: viewing data frames. Long term HTML support in GPUI is a big unknown to me.

[vultix]

@rgbkrk I have a working proof of concept of embedding a wry webview into GPUI. It's surprisingly simple, and surprisingly performant even with ~100 small webviews in a GPUI scrolling div.

First, add wry as a dependency in your Cargo.toml:

wry = "0.39.0"

Then, add this to the impl WindowContext in crates/gpui/src/window.rs

    /// Returns a reference to window's raw_window_handle::HasWindowHandle type
    pub fn raw_window_handle(&self) -> &dyn HasWindowHandle {
        &self.window.platform_window
    }

Finally, here's how I made a simple webview. Very kludgy, but works as a proof of concept:

use std::sync::Arc;

use gpui::*;
use wry::{dpi, Rect, WebView};
use wry::dpi::LogicalSize;
use wry::raw_window_handle::HasWindowHandle;

struct WebViewTest {
    views: Vec<Arc<WebView>>,
}

impl WebViewTest {
    fn new(num_views: usize, handle: &dyn HasWindowHandle) -> Self {
        let views = (0..num_views)
            .map(|i| {
                Arc::new(
                    wry::WebViewBuilder::new_as_child(&handle)
                        .with_html(format!(
                            "<html><body>Hello, world! I'm webview {i}</body></html>"
                        ))
                        .build()
                        .unwrap(),
                )
            })
            .collect();

        Self { views }
    }
}

impl Render for WebViewTest {
    fn render(&mut self, cx: &mut ViewContext<Self>) -> impl IntoElement {
        let mut parent = div()
            .id("parent")
            .block()
            .overflow_y_scroll()
            .size_full()
            .bg(rgb(0xff0000))
            .justify_center()
            .items_center();

        for (i, view) in self.views.iter().enumerate() {
            parent = parent.child(
                div()
                    .size(Length::Definite(DefiniteLength::Absolute(
                        AbsoluteLength::Pixels(Pixels(100.0)),
                    )))
                    .bg(rgb(0x00ff00))
                    .child(format!("This is webview {}:", i)),
            );
            parent = parent.child(HelloWorldEl { view: view.clone() });
        }

        parent
    }
}

struct HelloWorldEl {
    view: Arc<WebView>,
}
impl IntoElement for HelloWorldEl {
    type Element = HelloWorldEl;

    fn into_element(self) -> Self::Element {
        self
    }
}

impl Element for HelloWorldEl {
    type BeforeLayout = ();
    type AfterLayout = ();

    fn before_layout(&mut self, cx: &mut ElementContext) -> (LayoutId, Self::BeforeLayout) {
        let mut style = Style::default();
        style.flex_grow = 1.0;
        style.size = Size::full();
        let id = cx.request_layout(&style, []);
        (id, ())
    }

    fn after_layout(
        &mut self,
        bounds: Bounds<Pixels>,
        before_layout: &mut Self::BeforeLayout,
        cx: &mut ElementContext,
    ) -> Self::AfterLayout {
        // TODO: Find better way of detecting view visibility
        if bounds.top() > cx.viewport_size().height || bounds.bottom() < Pixels::ZERO {
            self.view.set_visible(false).unwrap();
        } else {
            self.view.set_visible(true).unwrap();

            self.view
                .set_bounds(Rect {
                    size: dpi::Size::Logical(LogicalSize {
                        width: (bounds.size.width.0 - 50.0).into(),
                        height: (bounds.size.height.0 / 2.0).into(),
                    }),
                    position: dpi::Position::Logical(dpi::LogicalPosition::new(
                        bounds.origin.x.into(),
                        bounds.origin.y.into(),
                    )),
                })
                .unwrap();
        }
    }

    fn paint(
        &mut self,
        bounds: Bounds<Pixels>,
        before_layout: &mut Self::BeforeLayout,
        after_layout: &mut Self::AfterLayout,
        cx: &mut ElementContext,
    ) {
        // Do nothing?
    }
}

fn main() {
    App::new().run(|cx: &mut AppContext| {
        let bounds = Bounds::centered(None, size(px(600.0), px(600.0)), cx);
        cx.open_window(
            WindowOptions {
                bounds: Some(bounds),
                ..Default::default()
            },
            |cx| {
                cx.activate_window();

                let view = WebViewTest::new(50, cx.raw_window_handle());
                cx.new_view(|_cx| view)
            },
        );
    });
}

[altaic]

Wondering if anyone has any experience with marimo (GitHub repo)? It looks really nice, with some benefits over Jupyter as discussed here.

[NathanielH-snek]

Wondering if anyone has any experience with marimo (GitHub repo)? It looks really nice, with some benefits over Jupyter as discussed here.

I'm also curious about this as well over jupyter notebooks, not having a whole different filetype in my workflows would be nice. Perhaps this could be an extension eventually?

[antaz]

I would like point out that it would be a good idea to make the REPL output scrollable, in the case of arbitrary long outputs, or truncate it somehow (Not sure what's the best solution)

[NyanHelsing]

@rgbkrk I have a working proof of concept of embedding a wry webview into GPUI. It's surprisingly simple, and surprisingly performant even with ~100 small webviews in a GPUI scrolling div.

First, add wry as a dependency in your Cargo.toml:

wry = "0.39.0"

Then, add this to the impl WindowContext in crates/gpui/src/window.rs

    /// Returns a reference to window's raw_window_handle::HasWindowHandle type
    pub fn raw_window_handle(&self) -> &dyn HasWindowHandle {
        &self.window.platform_window
    }

Finally, here's how I made a simple webview. Very kludgy, but works as a proof of concept:

use std::sync::Arc;

use gpui::*;
use wry::{dpi, Rect, WebView};
use wry::dpi::LogicalSize;
use wry::raw_window_handle::HasWindowHandle;

struct WebViewTest {
    views: Vec<Arc<WebView>>,
}

impl WebViewTest {
    fn new(num_views: usize, handle: &dyn HasWindowHandle) -> Self {
        let views = (0..num_views)
            .map(|i| {
                Arc::new(
                    wry::WebViewBuilder::new_as_child(&handle)
                        .with_html(format!(
                            "<html><body>Hello, world! I'm webview {i}</body></html>"
                        ))
                        .build()
                        .unwrap(),
                )
            })
            .collect();

        Self { views }
    }
}

impl Render for WebViewTest {
    fn render(&mut self, cx: &mut ViewContext<Self>) -> impl IntoElement {
        let mut parent = div()
            .id("parent")
            .block()
            .overflow_y_scroll()
            .size_full()
            .bg(rgb(0xff0000))
            .justify_center()
            .items_center();

        for (i, view) in self.views.iter().enumerate() {
            parent = parent.child(
                div()
                    .size(Length::Definite(DefiniteLength::Absolute(
                        AbsoluteLength::Pixels(Pixels(100.0)),
                    )))
                    .bg(rgb(0x00ff00))
                    .child(format!("This is webview {}:", i)),
            );
            parent = parent.child(HelloWorldEl { view: view.clone() });
        }

        parent
    }
}

struct HelloWorldEl {
    view: Arc<WebView>,
}
impl IntoElement for HelloWorldEl {
    type Element = HelloWorldEl;

    fn into_element(self) -> Self::Element {
        self
    }
}

impl Element for HelloWorldEl {
    type BeforeLayout = ();
    type AfterLayout = ();

    fn before_layout(&mut self, cx: &mut ElementContext) -> (LayoutId, Self::BeforeLayout) {
        let mut style = Style::default();
        style.flex_grow = 1.0;
        style.size = Size::full();
        let id = cx.request_layout(&style, []);
        (id, ())
    }

    fn after_layout(
        &mut self,
        bounds: Bounds<Pixels>,
        before_layout: &mut Self::BeforeLayout,
        cx: &mut ElementContext,
    ) -> Self::AfterLayout {
        // TODO: Find better way of detecting view visibility
        if bounds.top() > cx.viewport_size().height || bounds.bottom() < Pixels::ZERO {
            self.view.set_visible(false).unwrap();
        } else {
            self.view.set_visible(true).unwrap();

            self.view
                .set_bounds(Rect {
                    size: dpi::Size::Logical(LogicalSize {
                        width: (bounds.size.width.0 - 50.0).into(),
                        height: (bounds.size.height.0 / 2.0).into(),
                    }),
                    position: dpi::Position::Logical(dpi::LogicalPosition::new(
                        bounds.origin.x.into(),
                        bounds.origin.y.into(),
                    )),
                })
                .unwrap();
        }
    }

    fn paint(
        &mut self,
        bounds: Bounds<Pixels>,
        before_layout: &mut Self::BeforeLayout,
        after_layout: &mut Self::AfterLayout,
        cx: &mut ElementContext,
    ) {
        // Do nothing?
    }
}

fn main() {
    App::new().run(|cx: &mut AppContext| {
        let bounds = Bounds::centered(None, size(px(600.0), px(600.0)), cx);
        cx.open_window(
            WindowOptions {
                bounds: Some(bounds),
                ..Default::default()
            },
            |cx| {
                cx.activate_window();

                let view = WebViewTest::new(50, cx.raw_window_handle());
                cx.new_view(|_cx| view)
            },
        );
    });
}

This proof of concept is pretty cool and helpful

Figured out how to add a new output mime type here: https://github.com/zed-industries/zed/pull/17004/files was looking at wry like mentioned in the quoted above, also considered Servo;

Current task is figuring out how to render that webview into something from Zed's ui library. Looking at the quoted to figure that out.

[NyanHelsing]

@vultix I've started a draft PR based on your comment here, and it looks like the methods on Element that gpui is expecting has changed. I' trying to update the implementation to match that, but any pointers you can throw my way are appreciated!! See https://github.com/zed-industries/zed/pull/17004/files