jupyter-matplotlib 404

[cbcunc]

Filed at request of @parente to discuss at the Jupyter steering committee meeting next week.

Python 3.5. Dashboard Server bundle app using (from conda-forge) ipympl-0.0.2 and matplotlib-2.0.0b4 (required for ipympl) throws following in web console:

Error: Script error for "jupyter-matplotlib"
http://requirejs.org/docs/errors.html#scripterror

Matplotlib output not rendered (while ipywidgets output is). jupyter-dashboards-server reports on stdout:

GET /jupyter-matplotlib.js 404 5.874 ms - 675

jupyter kenelgateway reports on stdout:

[IPKernelApp] WARNING | Widget Javascript not detected.
It may not be installed properly.
Did you enable the widgetsnbextension?
If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"

Output of jupyter nbextension list:

(rsmbund) [cbc@bluheron bundle]$ jupyter nbextension list
Known nbextensions:
  config dir: /home/cbc/.jupyter/nbconfig
    notebook section
      jupyter-js-widgets/extension  enabled
      - Validating: OK
      jupyter-matplotlib/extension  enabled
      - Validating: OK
  config dir: /home/cbc/anaconda3/envs/rsmbund/etc/jupyter/nbconfig
    notebook section
      nb_anacondacloud/main  enabled
      - Validating: OK
      jupyter_dashboards/notebook/main  enabled
      - Validating: OK
      jupyter_cms/notebook/main  enabled
      - Validating: OK
      nb_conda/main  enabled
      - Validating: OK
      jupyter-js-widgets/extension  enabled
      - Validating: OK
      jupyter-matplotlib/extension  enabled
      - Validating: OK
      nbpresent/js/nbpresent.min  enabled
      - Validating: OK
    tree section
      jupyter_cms/dashboard/main  enabled
      - Validating: OK
      nb_conda/tree  enabled
      - Validating: OK
    edit section
      jupyter_cms/editor/main  enabled
      - Validating: OK

[parente]

Two possible solutions:

1. webpack the ipympl JS into the dashboard server itself like we do with jupyter-js-widgets for ipywidgets
2. get a bundler to include the ipympl JS alongside the dashboard notebook, and get whatever code in the notebook causes the load of the JS to get it from the path on the dashboard server (like we do with declarative widgets)

Longer term, maybe it would be nice to have a way to extend the dashboard server with arbitrary JS libs without sending PRs to it. But what that extension mechanism should look like is a giant ? to me at the moment.

/cc @SylvainCorlay

[parente]

@cbcunc Can you construct a minimal notebook that uses ipympl and evidences the problem without any external data as a test case? It'll give me a huge leg up in digging into what changes need to be made.

[cbcunc]

Sorry to take so long. Had to figure out how to get this to work in a simpler context. The minimal notebook with no external data can be found at https://gist.github.com/cbcunc/2e4edf4f73f0f7fc7ef9d4485e5b227c . An additional notebook which does the same thing, but without ipympl and the capability of separate figure and widget layout, can be found at: https://gist.github.com/cbcunc/3642675bbf1167008a29fac4d8a00cba .

[parente]

Thanks for creating the example. I'll see what I can do.

Note to self: https://github.com/matplotlib/jupyter-matplotlib/tree/master/js

[cbcunc]

Also, here's what I used for an environment. Notice the npm install f jupyter-matplotlib. I was not advised to install this at SciPy conference, although I wasn't advised not to either. But when I could not find jupyter-matplotlib.js, I decided to npm install it. It didn't help.

conda create -c conda-forge -n rsmbund jupyter
source activate rsmbund
conda install -y -c conda-forge/label/rc -c conda-forge h5py
conda install -y -c conda-forge/label/rc -c conda-forge libpng
conda install -y -c conda-forge/label/rc -c conda-forge matplotlib
conda install -c conda-forge/label/rc -c conda-forge ipywidgets
conda install -y -c conda-forge/label/rc -c conda-forge basemap
conda install -y -c conda-forge/label/rc -c conda-forge basemap-data-hires
conda install -y -c conda-forge/label/rc -c conda-forge owslib
pip install ipympl
jupyter nbextension install --py --symlink --user ipympl
jupyter nbextension enable --py --user ipympl
jupyter nbextension enable --py --sys-prefix --user widgetsnbextension
pip install jupyter_dashboards
jupyter dashboards quick-setup --sys-prefix
jupyter nbextension enable jupyter_dashboards --py --sys-prefix
pip install jupyter_cms
jupyter cms quick-setup --sys-prefix
pip install jupyter_dashboards_bundlers
jupyter dashboards_bundlers quick-setup --sys-prefix
pip install jupyter_kernel_gateway
npm install -g jupyter-dashboards-server
npm install -g jupyter-matplotlib

[parente]

Jotting down notes ...

For jupyter-js-widgets (the JavaScript code associated with ipywidgets), we:

1. Make it a dependency in package.json
2. Webpack it as part of the dashboard server build process (https://github.com/jupyter-incubator/dashboards_server/blob/master/gulpfile.js#L40)

For declarativewidgets, we:

1. Make sure the bundler picks up all of the bower-installed Polymer components at bundling time
2. Unbundle the notebook and those additional web assets into a directory on the dashboard server
3. Expose the web asset directory as a subpath of the dashboard notebook directory
4. Configure the declarative widgets to look in that subdirectory for its assets

For ipympl (and other libs), we have a couple options.

1. Give instructions about how users can add additional libraries to the webpack bundle so that they're available automatically, like ipywidgets. This is a pretty heavy-weight extension mechanism: it requires a rebuild of the dashboard server JS.
2. Give instructions on how users can bundle (manually or automatically) their additional JS assets and configure the library to find them under the proper dashboard server path. This is more flexible, but not all libraries are going to support configuration of where to find their assets. Many assume a hardcoded path in a notebook server. For example, it's not clear to me yet why the the dashboard example is trying to fetch from /jupyter-matplotlib.js. Why the root of the server?
3. @jhpedemonte, @dalogsdon other ideas?

[cbcunc]

One additional tidbit. I had heard some rumblings at SciPy conference that jupyter-mtaplolib.js was planned to moved out of jupyter-matplotlib and into matplotlib itself. It may be worth checking with @tacaswell and @SylvainCorlay on that.

[tacaswell]

@cbcunc Other direction, we are trying to move the js out of core mpl (which release at a glacial pace) and into it's own package which can release as frequently as needed.

[parente]

Leading into what I need to finish capturing on https://github.com/jupyter-incubator/dashboards_server/wiki/Widget-Support ...

The libraries that tend to work best in the dashboard server at the moment are those that emit their JavaScript as part of running code in the notebook (e.g., matplotlib 1.5.x, bokeh, plotly). That's because all of the necessary JS comes from code executing on the kernel as display_output and lands right on the page. Nothing needs to be installed out-of-band. And while, that's great for the dashboard use case (just run the notebook and voila!), it has its downsides. @tacaswell mentioned one: it tightly binds the JS to the kernel-side library.

[cbcunc]

Thanks @tacaswell . Good to know the drection of that library. @parente , I have jupyter-matplotlib installed globally via NPM. Maybe there's a way to have dashboard server ingest from the "out of band" global default, or specify it?

[jhpedemonte]

I was wondering if we could do something similar to the authentication plugins: have a default implementation which should work for simple libraries, but allow plugin code to be dropped into a directory of server -- plugin code would allow more complex setup of specific libraries.

Not sure if that setup would help with matplotlib here.

[parente]

Finally some time to dig into this. Good news, there's a quick-fix:

Need to step away, but will post back with the steps.

[parente]

# Create a conda environment with kernel gateway and nodejs in it
conda create -n ipympl-example python=3 nodejs jupyter_kernel_gateway
# Activate the environment
source activate ipympl
# Install ipympl from pypi
pip install ipympl
# Install the dashboard server (also into the environment, yay conda)
npm install -g jupyter-dashboards-server

# Make a directory to hold the dashboard example
mkdir -p /tmp/notebooks/example
# Put the notebook in the example directory as index.ipynb
cp with-ipympl.ipynb /tmp/notebooks/example/index.ipynb
# Tricky part: get the JS file out of the python package and dump it into the same folder
# as the dashboard notebook, renamed to match the require JS module sought
cp `python -c "import os; import ipympl; print(os.path.join(os.path.dirname(ipympl.__file__), 'static', 'index.js'))"` /tmp/notebooks/example/jupyter-matplotlib.js

# Fire up the kernel gateway in the background (or open another terminal)
jupyter kernelgateway &
# Fire up the dashboard server
jupyter-dashboards-server --NOTEBOOKS_DIR=/tmp/notebooks --KERNEL_GATEWAY_URL=http://127.0.0.1:8888

Then visit http://localhost:3000, click example, and the dashboard should load.

[cbcunc]

Best Christmas present ever!

[cbcunc]

With a couple of minor edits, I confirmed this is working. (Minor edits: choose either /tmp/dashboards or /tmp/notebooks as the server target, but not both :) Going now to test with the full blown storm surge app. Thanks!!!

[cbcunc]

Also, weird that I tried almost the same thing once before, at your suggestion, except that I copied jupyter-matplotlib.js from an npm install of jupyter-matplotlib instead of from ipympl and was using node from an installation in /usr instead of letting conda manage it.