Support javascript output

[GregorySchwartz]

I'm trying to use hvega which outputs vega-lite plots. The output is a javascript block. However, when running the source block I get no output and a warning: Warning (emacs): No valid mimetype found (:application/javascript). This is when trying the barcode example in https://raw.githubusercontent.com/DougBurke/hvega/master/notebooks/VegaLiteGallery.ipynb Obviously the interactivity isn't needed, but having it at least print out the javascript string would be beneficial on export. Best case scenario would be to have :file automatically export to .svg and .png.

MESSAGE: (:iopub :display-data (:data (:application/javascript requirejs({paths:{vg:’https://cdn.jsdelivr.net/npm/[email protected]/build/vega.min’,vl:’https://cdn.jsdelivr.net/npm/[email protected]/build/vega-lite.min’,vge:’https://cdn.jsdelivr.net/npm/[email protected]/build/vega-embed.min’},shim:{vge:{deps:[’vg.global’,’vl.global’]},vl:{deps:[’vg’]}}});define(’vg.global’,[’vg’],function(g){window.vega = g;});define(’vl.global’,[’vl’],function(g){window.vl = g;});var ndiv = document.createElement(’div’);ndiv.innerHTML = ’Awesome Vega-Lite visualization to appear here’;element[0].appendChild(ndiv);require([’vge’],function(vegaEmbed){vegaEmbed(ndiv,{"mark":"bar","data":{"values":[{"a":"A","b":28},{"a":"B","b":55},{"a":"C","b":43},{"a":"D","b":91},{"a":"E","b":81},{"a":"F","b":53},{"a":"G","b":19},{"a":"H","b":87},{"a":"I","b":52}]},"$schema":"https://vega.github.io/schema/vega-lite/v2.json","encoding":{"x":{"field":"a","type":"ordinal"},"y":{"field":"b","type":"quantitative"}},"description":"A simple bar chart with embedded data."}).then(function (result) { console.log(result); }).catch(function (error) { ndiv.innerHTML = ’There was an error: ’ + error; });});) :metadata nil))

[nnicandro]

I don't know how practical it would be to print out the javascript since it looks like vega-lite produces minified javascript and Emacs will slow down to be nearly unusable in the presence of long lines (http://lists.gnu.org/archive/html/emacs-devel/2018-10/msg00471.html). This will be an issue for complex plots.

Maybe we can do something like what is done in jupyter-widget-client.el and open up a browser to display the plot? But this isn't portable since the plot wouldn't be accessible from the org-mode document.

Exporting the plot to svg or png does look like a good option. I think we would still have to open up a browser to inject the javascript in. On a cursory look it should be possible to then send an SVG image string back to Emacs (https://github.com/vega/vega-view#view_toSVG) if we can get access to the Vega.View object in the div element that contains the plot (see https://github.com/vega/vega-embed#embed).

Alternatively, if there is a way to return the vega-lite JSON instead of Javascript, maybe we could use an external program to convert the plot into an svg or png file.

[GregorySchwartz]

I believe that's how altair does it https://altair-viz.github.io/user_guide/saving_charts.html, by sending the json to a browser and exporting either a png or an svg. Should this be done at the kernel level or the front-end (emacs-jupyter) level? I can see it both ways, as this is technically a front-end and thus should be displaying the correct information, but at the same time it might be easier for the backends to spit out the image. The benefit of converting json to an image when requested (if not specified, it should spit out formatted json I think), is that it won't be limited to vega-lite as it could be many things, maybe plotly, d3, and the like as well?

[GregorySchwartz]

Looking into it further, I think the front-end should handle this, as the kernel should just report the result (which is javascript).

[GregorySchwartz]

Would it be possible to use emacs' xwidget for this?

[nnicandro]

A long term goal of this package is to eventually use xwidgets for rendering javascript. But I am using the emacs-mac port on macOS which doesn't have good native support for the xwidget library (the closest is https://github.com/veshboo/emacs, but that is for the NS port and is not ready for primetime yet), so I haven't played around with xwidgets.

It does look like you could get it to work with xwidgets though since there are functions for evaluating javascript xwidget-webkit-execute-script. You should be able to get the example in your original post working by doing the following:

1. Create an html page, widget.html, that loads requirejs

<!DOCTYPE html>
<html>
<head>
<script type="application/javascript" src="https://requirejs.org/docs/release/2.3.6/minified/require.js"></script>
</head>
<body>
</body>
</html>

2. Browse to that page using xwidget

(xwidget-webkit-browse-url "widget.html")

3. Send code to the kernel that produces a plot and evaluate the returned javascript

(defun jupyter-xwidget-test ()
  (let ((code "<code that produces vega-lite plot>"))
    (jupyter-add-callback
        (jupyter-send-execute-request jupyter-current-client
          :code code)
      :display-data
      (jupyter-message-lambda ((res application/javascript))
        (when res
          (xwidget-webkit-execute-script
           (xwidget-webkit-last-session) res))))))

After you replace <code that produces vega-lite plot> with the example code and run step (2), you can call the above function while you are in the REPL buffer using M-: (jupyter-xwidget-test).

There is still the problem of getting at the Vega object in the div element that holds the plot so you can convert to an SVG string.

[UndeadKernel]

Another possibility would be to use Node.js to interpret the javascript and use its results. It would require the user to have it installed (which jupyter could test for and produce a warning if not found).

[nnicandro]

So to summarize what we have so far:

The issue is that there is currently no support for handling the application/javascript mimetype in the REPL or in org-mode source block results.

To handle application/javascript in org-mode source blocks we can:

1. Print the javascript in the org-mode buffer so that exporting to other formats can use it
   • This wouldn't work since printing minified Javascript (which we at least know vega-lite produces) would bump into the known issue of Emacs slowing down in the presence of long lines.
2. Send the javascript to an external browser and display it there
   • This would probably be one of the easiest to implement since we can modify the code in jupyter-widget-client.el and the javascript in the js/ directory to support evaluating arbitrary Javascript or use the skewer package.
3. Send the javascript to a headless browser, e.g. node.js or chrome's headless feature, convert to a PNG or SVG file, and send the image data back to Emacs
4. (Long term solution) Use Emacs' built-in xwidgets library to display the javascript directly in Emacs

[nnicandro]

With (2), (3), and (4) we still need to figure out how to preserve the results across sessions since the actual javascript used to produce the result would be lost. The only way to do this would be print the javascript in the buffer, but then we would probably run into the issue in (1).

Maybe we can have a different, hidden, buffer which stores the Javascript and is associated with the org-mode source block results. In the org-mode document we have a link which activates the display of the associated javascript when pressed. When the user closes the org-mode file, the link is replaced with the actual javascript. When the file is opened again, we extract the javascript in a hidden buffer and replace it with a link again. This way the user never sees the actual Javascript and movement in the org-mode buffer does not get slowed down by long lines.

Thoughts?

[GregorySchwartz]

The result is preserved as an svg or png, right? Like with all other plots, if something new is needed then the block is rerun. If you mean to have interactive javascript, then the javascript would be preserved as another type of separate file and "included" with org's include on export to html, but I first would recommend doing the svg and png to start simple.

[GregorySchwartz]

In that case, (3) would be best to start (using geckoengine or whatever altair does) and (4) would be long term interactive coolness (but not necessary if the export is html).

[nnicandro]

Yeah I was thinking that we could somehow preserve the javascript for exporting as well as generating the png or svg and using it as a stand-in for the real javascript, but I can't figure out a clean way of doing it without having to put the javascript in some temporary buffer so as to not bog down Emacs when it is minified. I was thinking we could do something with before-save-hook, after-save-hook, and find-file-hook to extract out javascript sections from the org buffer and write them back in when needed. It wouldn't be too complicated to implement, but it doesn't seem like the right approach.

It would be nice to keep all representations of a result around since that is what is done with notebooks, but there doesn't seem to be a way to do this in org-mode.

(3) does seem to be the best way to go. It looks like it wouldn't be too difficult to create bitmap images using a headless browser after looking at some examples:

• https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Headless_mode
• https://developers.google.com/web/updates/2017/04/headless-chrome

We can inject the javascript into an HTML page and run chrome --headless --screenshot ... or firefix --screenshot ... on the page to create a PNG image. We would probably have to figure out how to crop to the correct size of the image as well.

I think it would be harder to extract SVG images without knowing the library (vega, plotyly, ...) that created it. I'm not very fluent in javascript or web development so I don't really know how easy it is to traverse the DOM and find SVG images in a general way.

[lane-s]

I'm interested in helping with this, although in my case (using clojupyter with oz), it looks like the mime type of the response I'm getting in org-mode is html, but it has javascript embedded in the script tags. I'm not sure how we'd want to handle this, since it looks like there are other use cases for that response type besides displaying a chart. Maybe an option in the BEGIN_SRC header? Or we could check for a script tag.

As for saving the javascript that produced the chart, I have an idea that might allow us to avoid creating an extra emacs buffer. If the result of the source block is pointing to some svg or png file, maybe we could just create a js/html file with the same name. So you'd have blah.svg and blah.svg.js.

Since it seems like implementing png/svg support is probably the lowest hanging fruit for this, I'll start looking at doing that. For svg, I'll need to just try rendering with multiple chart libraries to see if there's a generic way to extract the rendered svg. For png, we'd need to do some kind of analysis to crop the full screenshot image to just the chart area.

If we're willing to introduce a node.js dependency for this, there are certainly some modules that would help. We could use puppeteer to start the headless browser, render the html/js, take a screenshot, then use sharp to process and crop the image. Otherwise, imagemagick has some auto-crop functionality we could use.

For example, I found that the following yields a decent image:

chrome --headless --disable-gpu --crash-dumps-dir=/tmp --screenshot=chart.png --virtual-time-budget=1000 file:///my_vega_output.html
convert chart.png -trim trimmed.png

One issue with doing it this way is that I don't see a way around doing multiple disk read/writes just to get the final image. I can't find any way to pass raw html into headless chrome, so it seems that we'd end up having to write an html file to pass to chrome, which would in turn write a png file that we would then pass to imagemagick in order to get the final image data.

Ideally we could just pipe the html data through a series of processes until we get the image. Using node.js+puppeteer+sharp would probably make this pretty easy, although if we do want to save the html anyway, then maybe sticking to the above commands is good enough.

I think we're pretty close on this. In summary, the key questions I've raised here are:

1. How should we recognize and handle charts that are embedded in html?
2. Can we store the js/html alongside the generated png/svg?
3. What do we want to use to crop images? Is it ok to use node.js?

In the meantime, I started working on this using node in the meantime since it seemed like the easiest path forward.

[gmoutso]

Perhaps panel is of interest? It might support vega. I have used it successfully with bokeh/holoviews. One way it should work with emacs-jupyter is via panel.panel(plot).show(port=8888, threaded=True). It then creates a bokeh server and popup a browser window. A new keyword arg to show (not yet in version 0.8) is show=False, which will inhibit popping up the window on remote servers (but can ssh forward the port and open it manually).

[nnicandro]

@lane-s Thanks for putting in the effort, and sorry for the delay in responding.

How should we recognize and handle charts that are embedded in html?

How about having a header argument, something like :eval-js yes, which enables the <script> block detection in html results and handles the result specially. :eval-js no (or no specification of :eval-js) will then mean to do what we do now even if <script> blocks exist in the html results.

Or something like :eval-html t to mean html results always get sent to a browser. Both header arguments my be needed to handle all cases.

Possibly a third option would be to, instead of generalizing, handle the clojure case specifically by adding something like

(defun jupyter-displayer-browser-p (data) ...)
(defun jupyter-display-browser (data) ...)

(cl-defmethod jupyter-org-result ((_mime (eql :text/html)) params data
                                  &context (jupyter-lang clojure)
                                  &optional _metadata)
  (if (jupyter-display-browser-p data)
      (jupyter-display-browser data)
    (cl-call-next-method)))

to a file jupyter-clojure.el (see the second paragraph in response to the third question below).

Note, for the <script> block detection, the following will be useful

(with-temp-buffer
  (insert "\
<html>
  <meta charset=\"utf-8\" />
  <title> Script in the Body </title>
  </head>
  <body>
  <script type = \"text/javascript\">
  1 + 1 
  </script>
  </body>
</html>
")
  (dom-by-tag 
   (libxml-parse-html-region (point-min) (point-max))
   'script))

Can we store the js/html alongside the generated png/svg?

My comments were related to how we should handle the fact that plotting libraries like Vega will return application/javascript (or text/html with JavaScript embedded) that could potentially contain very long lines of minified JavaScript. For example this notebook contains the following line

"const spec = {\"$schema\": \"https://vega.github.io/schema/vega/v5.json\", \"width\": 400, \"height\": 200, \"padding\": 5, \"data\": [{\"name\": \"table\", \"values\": [{\"category\": \"A\", \"amount\": 28}, {\"category\": \"B\", \"amount\": 55}, {\"category\": \"C\", \"amount\": 43}, {\"category\": \"D\", \"amount\": 91}, {\"category\": \"E\", \"amount\": 81}, {\"category\": \"F\", \"amount\": 53}, {\"category\": \"G\", \"amount\": 19}, {\"category\": \"H\", \"amount\": 87}]}], \"signals\": [{\"name\": \"tooltip\", \"value\": {}, \"on\": [{\"events\": \"rect:mouseover\", \"update\": \"datum\"}, {\"events\": \"rect:mouseout\", \"update\": \"{}\"}]}], \"scales\": [{\"name\": \"xscale\", \"type\": \"band\", \"domain\": {\"data\": \"table\", \"field\": \"category\"}, \"range\": \"width\", \"padding\": 0.05, \"round\": true}, {\"name\": \"yscale\", \"domain\": {\"data\": \"table\", \"field\": \"amount\"}, \"nice\": true, \"range\": \"height\"}], \"axes\": [{\"orient\": \"bottom\", \"scale\": \"xscale\"}, {\"orient\": \"left\", \"scale\": \"yscale\"}], \"marks\": [{\"type\": \"rect\", \"from\": {\"data\": \"table\"}, \"encode\": {\"enter\": {\"x\": {\"scale\": \"xscale\", \"field\": \"category\"}, \"width\": {\"scale\": \"xscale\", \"band\": 1}, \"y\": {\"scale\": \"yscale\", \"field\": \"amount\"}, \"y2\": {\"scale\": \"yscale\", \"value\": 0}}, \"update\": {\"fill\": {\"value\": \"steelblue\"}}, \"hover\": {\"fill\": {\"value\": \"red\"}}}}, {\"type\": \"text\", \"encode\": {\"enter\": {\"align\": {\"value\": \"center\"}, \"baseline\": {\"value\": \"bottom\"}, \"fill\": {\"value\": \"#333\"}}, \"update\": {\"x\": {\"scale\": \"xscale\", \"signal\": \"tooltip.category\", \"band\": 0.5}, \"y\": {\"scale\": \"yscale\", \"signal\": \"tooltip.amount\", \"offset\": -2}, \"text\": {\"signal\": \"tooltip.amount\"}, \"fillOpacity\": [{\"test\": \"datum === tooltip\", \"value\": 0}, {\"value\": 1}]}}}]};\n"

If the lines are very long Emacs will be noticabley sluggish.

My idea to counteract this was to, after receiving an application/javascript result that the user wants to display, copy over the raw JavaScript to a buffer and have the displayed result be something like

#+RESULTS:
[[eval:jupyter-org-display-result-in-browser][Open browser]]

Then the user can click on the link to actually display the browser showing the JavaScript.

I would still like to ensure that the Org document remains portable, so when a user saves it, the original application/javascript would be saved as part of the Org document, so instead of a link above, something like the following would be saved

#+RESULTS:
#+begin_example js
...
#+end_example

(We may not actually need to replace the JavaScript with a link. We may be able to overlay (https://www.gnu.org/software/emacs/manual/html_node/elisp/Overlays.html) to mitigate any issues with long lines.)

One way to do this would be all the buffer manipulation stuff I mentioned.

Copying the JavaScript over to file would be fine for processing it into an image, but how would we make the document portable?

What do we want to use to crop images? Is it ok to use node.js?

I'm not sure I would like to introduce a dependency outside of Emacs specifically for this case since only a subset of users would use it. Maybe have the chromium -> convert way be a default and the node.js way be used if node is available?

I plan on splitting out the kernel language dependent parts of this project into separate sub-projects under the emacs-juyter organization, see #227. We could then make an emacs-jupyter/jupyter-clojure repo that can have as many dependencies as it needs for clojupyter users.

[wuqui]

Is there any way of rendering altair plots inline just like matplotlib plots?

I'm using two workarounds, which are okay, but not perfect.

1. Letting altair display the results in the browser by using:

import altair_viewer
alt.renderers.enable('altair_viewer')

2. Saving plots using altair_saver and then manually inserting the resulting images in the org buffer after the empty results block.

It seems altair is supporting several rendering options, but none of them worked for me. Unfortunately, I know very little about these things.

Is there a better way to do it? Hope this is the right place to ask...

[dimitar-petrov]

There is a solution that worked for me.

alt.renderers.enable('altair_saver', fmts=['png'], embed_options={'scaleFactor': '1'})

Make sure that you install altair_saver properly with all external dependencies.

• selenium
• npm packages