Support multiple `pyodide` kernelspecs

[bollwyvl]

Problem

At present, the server extension registers a single kernelspec, and each kernel session is started with exactly that configuration.

In looking at supporting pyodide-lock (a la #49), a number of techniques currently in use, such as maintaining multiple versions of patched upstreams in the Warehouse all.json, will no longer work for the general case.

As a single pyodide-lock.json can only include one (or zero) version of each named package. The "interactive" solver will have less opportunity to load the "correct" package for corner cases, especially for a general purpose interactive site (as opposed to a well-engineered environment to complement some learning goals). Consider:

Open two notebooks, side-by-side, with two versions of the same package installed via

notebook 1	notebook 2
%pip install some-package=x.y	%pip install some-package=x.z
import some_package	import some_package
%timeit some_package.some_func()	%timeit some_package.some_func()

Compare the run times.

Proposed Solution

Offer a way to more deeply configure the default python kernlspec, and create extra named kernelspecs which could have separate:

• labels, e.g.
  • python = Python (Pyodide 0.25.0)
  • python-custom = Python (Pyodide 0.25.0 + my-site)
• launcher icons
• loadPyodide configuration, e.g.
  • environment variables
  • pyodide-lock.json (or none at all, using the "stock" one provided with pyodide.js)
  • python arguments
    • can include filterwarnings, etc.
  • packages to load
    • but only if
    • pyodide-lock.json is provided
    • it is "complete" including ipython and dependencies

Additional context

For site users

• the pyodide-lock.json approach is probably a better UX
• multiple kernelspecs will make things a bit noisier/more crowded

For site owners

• the pyodide-lock.json approach is strictly more reproducible
• unlike all.json, a pyodide-lock.json can't reference PyPI-hosted dependencies, so a full solve is required
• there is no lightweight way to get an authoritative micropip.freeze solve from a standard CPython build environment
  • at present, the most reliable approach is likely to
    • open a fresh browser
    • piplite.install all packages on top of the baseline
    • capture the output of a new piplite.freeze, as we may need to do some rewriting
• the CLI can probably only handle operating against the python kernelspec
  • successive kernelspecs would likely need to be configured at build time through a new well-known location, e.g.
  • `{lite_dir}/pyodide-kernelspecs/
    • {name}/
      • kernelspec.json
  • the names of these would be injected into jupyter-config-data, and lazily resolved

Out of scope

• an even more extreme approach could allow an .ipynb to offer its own kernelspec overrides
  • this is probably a bridge too far, but leaving a hook for this to be accomplished by an extension would be possible, especially if it worked for many kernels
• multiple kernelspecs could in turn support multiple versions of pyodide on the same site
  • this would increase maintenance burden, as each release would need to test against a matrix of supported versions instead of exactly one, losing robust typescript typing along the way, among other things
  • the as-deployed structure would likely have to change from /static/pyodide to `/static/pyodide/{version-id}/

[juntyr]

+1. Perhaps a simpler way to experiment with the userside functionality would be an optional URL parameter for changing the pyodide-lock.json file for all kernel launches on a page. This would still just offer two options per tab (override default or no override) but would already allow users to start with a custom pre-solved environment.

[bollwyvl]

It's... very hard to make a functional pyodide-lock.json that will work over time for a given site.