[pydeck] Create a prebuilt extension for JupyterLab

[dakoop]

For #6989

The general idea is to create a prebuilt extension that can be distributed as part of the pydeck python package. There is no need for a separate @deck.gl/jupyter-widget install as it is bundled in the pydeck python package, and the extension does not require jupyterlab to be rebuilt as it uses webpack 5's module federation. Just run pip install pydeck. This is WIP as it doesn't currently check for compatibility with classic notebook support, etc. To build,

1. yarn bootstrap
2. Install build and jupyter_packaging for python
3. Run python -m build in bindings/pydeck a. If this doesn't work, try jlpm build in modules/jupyter-widget b. This should create a wheel in dist
4. pip install dist/*.whl

Change List

• Integrates @jupyterlab/builder and webpack 5 into the jupyter-widget build process. This is a one-step build now as the jupyter labextension build uses webpack to create a federated module (no separate dist/index.js)
• Removes acorn resolution in the root package.json. Some part of the new build process requires a newer version of acorn, as the build just hangs otherwise. I am not sure what the effect is on other parts of deck.gl.
• Writes the generated labextension to pydeck/labextension in bindings/pydeck so it can be distributed as part of the python wheel.
• Supports the python -m build method for building pydeck and the labextension
• Updates some metadata suggested in jupyterlab extensions
• Updates pydeck version to 0.8.0
• Builds on #7026

Todo

• [ ] Fix issue with jupyter labextension develop . (something with pyproject.toml?)
• [ ] Check whether other pieces of pydeck and jupyter-widget function as before
• [ ] Determine impact of acorn resolution
• [ ] Fix other pydeck setup/build code (e.g. Makefile)
• [ ] Update documentation

[kylebarron]

install as it is bundled in the pydeck python package

iirc this is only valid for specific jupyter versions. Only for notebook as of version 5.3 and Jupyter lab as of version 3.

[dakoop]

Correct, although even notebook will now be built on jupyterlab infrastructure (ref). I think you could keep the other method (current install process) working for the other modes, but these would have to be tested. JupyterLab developers are aware of some difficulties for users finding and installing the new prebuilt extensions: https://github.com/jupyterlab/jupyterlab/issues/10554

Perhaps a better approach in structuring the code would be to structure the jupyter-widget components as currently done, and create a separate build process for the JupyterLab 3 plugin so that different dependency versions (specific to JupyterLab's build/plugin process) could be integrated without impacting the deck.gl's current dependencies. I didn't end up going that route because I couldn't figure out what was going on with the original approach that just included dist/index.js, but in retrospect, this may have been due to the dependency issues (acorn).

[davidbrochart]

@dakoop do you plan on continuing that work? It would be awesome.

[donmccurdy]

My impression (not a regular Jupyter user!) is that nbextensions for Jupyter are dying a slow death at this point, and with Jupyter Notebooks v7+ (released July 2023), you'll get errors like ...

Jupyter command `jupyter-nbextension` not found.

... for which old solutions will no longer work. Likely a prebuilt extension for JupyterLab is becoming necessary. Are there other alternatives we should consider, or anything I can do to help land this PR?

Otherwise, I think pydeck may be stuck in Jupyter Notebook 6.4 or lower, or nbclassic.

[kylebarron]

I think the main difference is that Jupyter Notebook v7 is built upon JupyterLab architecture, but in a way that looks like the classic notebook. So I think you need to make JupyterLab extensions to support the latest notebook

[dakoop]

As @kylebarron noted, Notebook 7 introduces extension changes as it moved to lab architecture. This PR was a push to update the pydeck extension a while back, but I haven't had time to reexamine this. There is other work on deck.gl Jupyter integration: lonboard and ipydeck.