Pluto integration

[fonsp]

trafficstars

Hi! It would be great if Pluto notebooks could work as an input format for Quarto! I talked briefly about this with @MichaelHatherly and with the Quarto devs at JupyterCon last year :) I believe that quite a nice integration is possible!

This issue could be used to collect ideas on what the integration would look like, and to find out whether there is interest in this feature.

[co1emi11er2]

Yes this would be an awesome integration! As someone who sees the potential use for this in my area of industry, let me know if I can help in any way.

[MichaelHatherly]

Hi! It would be great if Pluto notebooks could work as an input format for Quarto!

What do you have in mind with this integration? This backend just takes .qmds (or quarto's script format, https://quarto.org/docs/computations/render-scripts.html) and runs them. By integration are you meaning

quarto render path/to/pluto/notebook.jl

and have that "just work" for .jl files produced by Pluto?

[co1emi11er2]

I can take a shot at an idea. I don't know how Pluto, QuartoNotebookRunner (lets abreviate this to QNR), or Quarto work under the hood, so this could be completely wrong.

First question would be: If I had a pluto notebook that I kept linear in terms of cell order and stuck to the quarto script format, would this be rendered correctly already with QNR?

Idea

This is under the assumption that quarto passes the whole script to QNR and QNR does the parsing of the file. If quarto passes the cell chunks to QNR to then run, then this idea breaks down.

My idea would be:

1. Tell QNR that it is a pluto notebook.

It could be QNR determining that on its own by looking for the ### A Pluto.jl notebook ### or # ╔═╡ type symbols in the file. Or something in the yaml header block.

2. QNR would then let Pluto or PlutoDependencyExplorer to parse the file.

Hopefully we can keep the comments within the cell blocks to keep quarto commands so we can adjust outputs. I know macros parse comments out so it may have to be parsed back in afterwards depending on how pluto works.

By parsing the file I imagine having a list of code, output that is sorted not by the linear order needed to run the code, but by the order specified by how you see it rendered in the pluto document. I imagine that tooling is already there within pluto.

3. Parsed input/outputs passed to quarto to render

Now with this list quarto can then take that information to build the pdf/html/etc.

[MichaelHatherly]

This is under the assumption that quarto passes the whole script to QNR and QNR does the parsing of the file.

That is correct, quarto (the binary) passes the file contents as a whole to our backend, which then sends the cells to worker processes for evaluation.

The main question we need to address is still https://github.com/PumasAI/QuartoNotebookRunner.jl/issues/125#issuecomment-2098650720, what do you expect from the high-level user-interface for this? Just run quarto render directly on a Pluto notebook @co1emi11er2? Let's not worry about all the internal workings of everything just yet.

[co1emi11er2]

I guess the overall goal for me is to make it even more high level (no CLI). I want just an export or preview button inside pluto sort of like vscode extension where you can see the preview and saving the file will auto-render it again. But I would be fine with quarto render as the starting point for sure.

Most of the people I hope to convince to use pluto do not know the terminal, julia, quarto, or python. Our main tool for calculations is excel, so the more things I can pull away from a CLI, the better.

[MichaelHatherly]

Thanks. If this is just about getting nice Quarto renders of Pluto notebooks (which would be great) then the most straightforward way would be a feature directly in Pluto that uses quarto_jll.jl to shell out to quarto render and provides it as input a fully executed markdown/ipynb file that quarto already understands how to render.

That's pretty much what this package does internally anyway. We generate JSON output for executed .qmd in .ipynb format that quarto render already knew how to render.

[MichaelHatherly]

It would be great if Pluto notebooks could work as an input format for Quarto!

This one is a different request. Doing something like that requires first making quarto understand Pluto's notebook format. That is not actionable in this package. It'll need upstream changes.

[robot144]

For my work teaching, it would be great if topic explanations written as pluto files could be integrated into a website generated with quarto. The pluto files will then contain interactivity which will be rendered to html, so that the explanations can be read and figures seen. For interactivity, or even playing with the code, it would be great to have a link to the pluto file that can be pasted into a running pluto instance. Or even a link that opens the file in binder for example.

[younes-io]

is this abandoned? I think this would be really great! I've been waiting for this for months now and I hope it could become concrete some day..

Marimo (which is mainly inspired by Pluto) is working on something similar: https://docs.marimo.io/guides/exporting.html#embed-marimo-outputs-in-html-using-islands

Demo's here: https://docs.marimo.io/guides/island_example.html

[MichaelHatherly]

is this abandoned? I think this would be really great! I've been waiting for this for months now and I hope it could become concrete some day..

It isn't abandoned, just waiting on someone that wants this to take the time to do something about it. As in my previous comments, most of what's being requested here needs to be implemented upstream unless I'm misunderstanding what others have requested.

[jkrumbiegel]

Just adding a comment here that what was discussed further above, that quarto passes the input document as-is to QuartoNotebookRunner, is not quite true. There are shortcodes and other things that may modify the input markdown a bit before it reaches us. So quarto in general needs to be able to analyze the input, too, so it cannot just be something completely special that only QuartoNotebookRunner knows how to handle.

[VarLad]

I'm not sure how efficient this is, but I think the goal here is to provide a good user experience in Pluto for editing Quarto. For that, at least to me, the best way forward looks like:

1. pluto_notebook.jl -> notebook.qmd (using Julia and FileWatching for every update the user makes in Pluto)
2. notebook.qmd -> pdf/html (whatever the user desires, using quarto preview, we leave it to the user. This will also automatically trigger QuartoNotebookRunner to run the Julia cells)

For the user experience, we can have a package, which gives us a few things:

1. pandoc markdown (supported by Pandoc.jl) for consistency Something like this:

using Pandoc

struct QuartoString
    s::String
end

struct YAMLString
    y::String
end

macro yaml_str(s)
    quote
        YAMLString(s)
    end
end

macro qmd_str(s)
	quote
  	  p = run(Pandoc.Converter(;input = @raw_str($s), mathml = true))
	  QuartoString(p)
	end
end

function Base.show(io::IO, m::MIME"text/html", qs::QuartoString)
    show(io, m, HTML(qs.s))
end

function Base.show(io::IO, m::MIME"text/html", ys::YAMLString)
    show(io, m, HTML(@md_str("```yaml\n" * ys.s * "\n```"))
end

• qmd"" and yaml"" cells in Pluto would be converted to markdown and yaml respectively.
• All other cells get converted to:

```{julia}

```

blocks in qmd

2. An option to embed the HTML preview. (in the right half of the screen preferably)
3. We skip the cell that imports the package. Also we assert that the first cell is a YAML String cell.

This should be good enough to provide a good user experience for working with Quarto, directly from Pluto.jl. I think most of the components are there already but we need a Pluto parser for this or something similar.

I'd like to hear everyone's opinions on this.

[VarLad]

Its seems that directly converting Pluto.jl notebook to ipynb would be more effective way than converting to qmd first.

For people who know more about Pluto, is there a way to get the output of cells of a running Pluto notebook?

[fonsp]

Nice that you're looking into this @VarLad! Since you want cell output of a running notebook, you need to hook into the live Pluto server. Look into PlutoEvent (check out our source code). You probably want to subscribe to StateChangeEvent (for any update) or FileSaveEvent or both.

Both events give you the running Notebook as event argument. You can do notebook.cells to get the cells, and then cell.output etc (check out our source code). Not all Pluto cells output plain text or html, some have rich objects like the object tree viewer of rand(10). Check out https://github.com/rikhuijzer/PlutoStaticHTML.jl for a project that solves this problem. I also wrote a script once that converts a notebook (with outputs) to Markdown: https://github.com/fonsp/Promises.jl/blob/main/tools/generate_readme2.jl

---

A completely different approach is to not use the Pluto package, but implement this using the notebook file and statefile. In that case you could also write this in TS, which means you can integrate with Quarto directly. (Then you could also support Pluto UI like the object tree viewer, @bind, pretty stack traces and more). For this, take a running notebook URL and replace edit with notebookfile for the .jl file and statefile for the MsgPack state blob. You can unpack the state blob with Julia (Pluto.unpack) or JS.