jupyterlite
FileSystem calls over Atomics.wait instead of service worker
In combination with https://github.com/jupyterlite/jupyterlite/pull/1383
This PR provides an implementation of the Emscripten FileSystem that communicates using Atomics.wait and SharedArrayBuffer instead of the blocking HTTP requests to a service worker.
~~I open a PR here in jupyterlite/xeus for convenience, but part of the code here should probably go upstream in jupyterlite.~~ Done in https://github.com/jupyterlite/jupyterlite/pull/1383
How to test it:
- Build your jupyterlite site as usual.
- Serve it using the proper headers, so that shared array buffers can be used:
npx static-handler --cors --coop --coep --corp ./
Implementation details
Instead of directly using SharedArrayBuffers and Atomic.waits (which would require defining a proper communication protocol), I decided to go with using https://github.com/WebReflection/coincident which provides high level APIs for making blocking communication between worker <-> main thread.
Coincident comes as a replacement for comlink which we used for wrapping the worker kernel communication.
I'll turn this into draft and leave it open for discussions. But I'll open the proper PR upstream in JupyterLite to introduce the needed APIs for this to work nicely.
This is all working nicely (except ipyleaflet of course, because CORS are disabled...)
We should probably document how to enable this. It could be here in the jupyterlite-xeus docs, or maybe in the main JupyterLite documentation directly, as we will probably want to support it in the Pyodide kernel too?
We should definitely look at enabling this for jupyterlite-pyodide too :)
Agreed we should add it to the docs.
Testing locally: one advantage of this approach is that it works in a Firefox private window :+1:
This should be backward compatible if not setting --cors --coop --coep --corp (actually the CI passed because it's backward compatible in that case, we're still using jupyterlite 0.3.x there).
If people were already setting --cors --coop --coep --corp, file access will eventually fail and crash the kernel due to https://github.com/jupyterlite/xeus/pull/87/files#diff-a0841713a169cff378e9935d6ae3ba34d57653c34fcea3294ac4771db5427b14R185
So I believe this is backward incompatible.
I believe we can't make it backward compatible though, so we should probably make an alpha for 0.2.0. Would you like to do it ?
Yeah I was thinking a 0.2.0 would be on the safer side too.
Starting a 0.2.0-alpha.0 now.
We should probably document how to enable this. It could be here in the
jupyterlite-xeusdocs, or maybe in the main JupyterLite documentation directly,
Started to document this in https://github.com/jupyterlite/jupyterlite/pull/1405
I can have a look at implementing the shared array buffers approach for the pyodide kernel.
I can have a look at implementing the shared array buffers approach for the pyodide kernel.
That would be great, thanks! Hopefully the approach will be similar.