jupyter-server-proxy
jupyter-server-proxy copied to clipboard
Windows support
When trying to run jupyterlab-nvdashboard on windows I'm getting the following error on startup.
(Sorry about the screenshot, copying from powershell seems to lose all linebreaks)
I imagine this is related to the way the server proxy is using asyncio/tornado to create subprocesses.
Conda env
name: pipeline
channels:
- conda-forge
- defaults
dependencies:
- attrs=19.1.0=py_0
- backcall=0.1.0=py_0
- bleach=3.1.0=py_0
- bokeh=1.3.4=py37_0
- ca-certificates=2019.5.15=1
- certifi=2019.6.16=py37_1
- click=7.0=py_0
- cloudpickle=1.2.2=py_0
- colorama=0.4.1=py_0
- cudatoolkit=10.0.130=0
- cudnn=7.6.0=cuda10.0_0
- cupy=6.0.0=py37h230ac6f_0
- cytoolz=0.10.0=py37hfa6e2cd_0
- dask=2.4.0=py_0
- dask-core=2.4.0=py_0
- decorator=4.4.0=py_0
- defusedxml=0.5.0=py_1
- distributed=2.4.0=py_0
- entrypoints=0.3=py37_1000
- fastrlock=0.4=py37h6538335_1000
- freetype=2.10.0=h563cfd7_1
- fsspec=0.5.1=py_0
- heapdict=1.0.0=py37_1000
- intel-openmp=2019.5=281
- ipykernel=5.1.2=py37h5ca1d4c_0
- ipython=7.8.0=py37h5ca1d4c_0
- ipython_genutils=0.2.0=py_1
- jedi=0.15.1=py37_0
- jinja2=2.10.1=py_0
- jpeg=9c=hfa6e2cd_1001
- json5=0.8.5=py_0
- jsonschema=3.0.2=py37_0
- jupyter_client=5.3.1=py_0
- jupyter_core=4.4.0=py_0
- jupyterlab=1.1.4=py_0
- jupyterlab_server=1.0.6=py_0
- libblas=3.8.0=12_mkl
- libcblas=3.8.0=12_mkl
- liblapack=3.8.0=12_mkl
- libpng=1.6.37=h7602738_0
- libsodium=1.0.17=h2fa13f4_0
- libtiff=4.0.10=h6512ee2_1003
- llvmlite=0.29.0=py37hed17590_1
- locket=0.2.0=py_2
- lz4-c=1.8.3=he025d50_1001
- m2w64-gcc-libgfortran=5.3.0=6
- m2w64-gcc-libs=5.3.0=7
- m2w64-gcc-libs-core=5.3.0=7
- m2w64-gmp=6.1.0=2
- m2w64-libwinpthread-git=5.0.0.4634.697f757=2
- markupsafe=1.1.1=py37hfa6e2cd_0
- mistune=0.8.4=py37hfa6e2cd_1000
- mkl=2019.4=245
- msgpack-python=0.6.2=py37he980bc4_0
- msys2-conda-epoch=20160418=1
- nbconvert=5.6.0=py37_1
- nbformat=4.4.0=py_1
- nodejs=10.13.0=0
- notebook=6.0.1=py37_0
- numba=0.45.1=py37hf9181ef_0
- numpy=1.15.4=py37h8078771_1002
- olefile=0.46=py_0
- openssl=1.1.1d=he774522_0
- packaging=19.2=py_0
- pandas=0.25.1=py37he350917_0
- pandoc=2.7.3=0
- pandocfilters=1.4.2=py_1
- parso=0.5.1=py_0
- partd=1.0.0=py_0
- pickleshare=0.7.5=py37_1000
- pillow=6.1.0=py37h643dfcc_1
- pip=19.2.3=py37_0
- prometheus_client=0.7.1=py_0
- prompt_toolkit=2.0.9=py_0
- psutil=5.6.3=py37hfa6e2cd_0
- pygments=2.4.2=py_0
- pyparsing=2.4.2=py_0
- pyrsistent=0.15.4=py37hfa6e2cd_0
- python=3.7.3=h510b542_1
- python-dateutil=2.8.0=py_0
- pytz=2019.2=py_0
- pywinpty=0.5.5=py37_1000
- pyyaml=5.1.2=py37hfa6e2cd_0
- pyzmq=18.1.0=py37h16f9016_0
- send2trash=1.5.0=py_0
- setuptools=41.2.0=py37_0
- six=1.12.0=py37_1000
- sortedcontainers=2.1.0=py_0
- sqlite=3.29.0=hfa6e2cd_1
- tbb=2019.8=he980bc4_0
- tblib=1.4.0=py_0
- terminado=0.8.2=py37_0
- testpath=0.4.2=py_1001
- tk=8.6.9=hfa6e2cd_1003
- toolz=0.10.0=py_0
- tornado=6.0.3=py37hfa6e2cd_0
- traitlets=4.3.2=py37_1000
- vc=14.1=h0510ff6_4
- vs2015_runtime=14.16.27012=hf0eaf9b_0
- wcwidth=0.1.7=py_1
- webencodings=0.5.1=py_1
- wheel=0.33.6=py37_0
- wincertstore=0.2=py37_1002
- winpty=0.4.3=4
- xz=5.2.4=h2fa13f4_1001
- yaml=0.1.7=hfa6e2cd_1001
- zeromq=4.3.2=h6538335_2
- zict=1.0.0=py_0
- zlib=1.2.11=h2fa13f4_1006
- zstd=1.4.0=hd8a0e53_0
- pip:
- aiohttp==3.6.1
- async-timeout==3.0.1
- chardet==3.0.4
- idna==2.8
- jupyter-server-proxy==1.1.0
- jupyterlab-nvdashboard==0.1.9
- multidict==4.5.2
- pynvml==8.0.3
- simpervisor==0.3
- yarl==1.3.0
prefix: C:\Users\jacob\Miniconda3\envs\pipeline
Thanks for the report @jacobtomlinson. Yes, it seems that some asyncio features used by simpervisor are not supported on Windows. I don't know if this can be addressed within simpervisor -- that's out of my depth. I think the most we can do here is document this issue.
I came across the same issue on windows, and made an issue in the simpervisor
repo.
I did a bit more research on the windows support, my conclusion is that we may need to implement a fallback solution for windows which does not use simpervisor.
The reason is:
The default asyncio event loop implementation on Windows does not support subprocesses. Subprocesses are available for Windows if a ProactorEventLoop is used. See Subprocess Support on Windows for details.
https://docs.python.org/3/library/asyncio-subprocess.html
I tried to switch the eventloop for simpervisor:
if sys.platform == 'win32':
loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)
However, the entire jupyter server hangs, and I figured it maybe because tornado server doesn't not support ProactorEventLoop
:
https://github.com/tornadoweb/tornado/blob/d6819307ee050bbd8ec5deb623e9150ce2220ef9/tornado/platform/asyncio.py#L15-L19
We may need to implement a synchronous version of the SupervisedProcess, and run it in a thread/executor instead.
@jacobtomlinson I made a PR for this issue, would be great if you can do a test by running pip install -U http://github.com/oeway/jupyter-server-proxy/tarball/master#egg=jupyter-server-proxy
?
@oeway Tried this, does not work with nvdashboard.
After pip install -U http://github.com/oeway/jupyter-server-proxy/tarball/master#egg=jupyter-server-proxy
, I even build jupyter labextension install jupyterlab-nvdashboard
Here is the stacktrace
Traceback (most recent call last):
File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\web.py", line 1703, in _execute
result = await result
File "C:\Users\mayan\anaconda3\lib\site-packages\jupyter_server_proxy\websocket.py", line 97, in get
return await self.http_get(*args, **kwargs)
File "C:\Users\mayan\anaconda3\lib\site-packages\jupyter_server_proxy\handlers.py", line 534, in http_get
return await self.proxy(self.port, path)
File "C:\Users\mayan\anaconda3\lib\site-packages\jupyter_server_proxy\handlers.py", line 528, in proxy
await self.ensure_process()
File "C:\Users\mayan\anaconda3\lib\site-packages\jupyter_server_proxy\handlers.py", line 505, in ensure_process
await proc.start()
File "C:\Users\mayan\anaconda3\lib\site-packages\simpervisor\process.py", line 90, in start
self.proc = await asyncio.create_subprocess_exec(
File "C:\Users\mayan\anaconda3\lib\asyncio\subprocess.py", line 236, in create_subprocess_exec
transport, protocol = await loop.subprocess_exec(
File "C:\Users\mayan\anaconda3\lib\asyncio\base_events.py", line 1630, in subprocess_exec
transport = await self._make_subprocess_transport(
File "C:\Users\mayan\anaconda3\lib\asyncio\base_events.py", line 491, in _make_subprocess_transport
raise NotImplementedError
NotImplementedError
@oeway My bad, it works!
After pip install -U http://github.com/oeway/jupyter-server-proxy/tarball/master#egg=jupyter-server-proxy
Just run jupyter lab
via the terminal, no need to build anything. The ProactorEventLoop takes care of it. 💯
There are a couple of uncaught exceptions thrown by the tornado web-server which I am listing here:
[E 17:57:37.908 LabApp] Uncaught exception
Traceback (most recent call last):
File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\websocket.py", line 649, in _run_callback
result = callback(*args, **kwargs)
File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\websocket.py", line 1528, in on_message
return self._on_message(message)
File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\websocket.py", line 1534, in _on_message
self._on_message_callback(message)
File "C:\Users\mayan\anaconda3\lib\site-packages\jupyter_server_proxy\handlers.py", line 298, in message_cb
self.write_message(message, binary=isinstance(message, bytes))
File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\websocket.py", line 339, in write_message
raise WebSocketClosedError()
tornado.websocket.WebSocketClosedError
[E 17:57:37.990 LabApp] Uncaught exception
Traceback (most recent call last):
File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\websocket.py", line 649, in _run_callback
result = callback(*args, **kwargs)
File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\websocket.py", line 1528, in on_message
return self._on_message(message)
File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\websocket.py", line 1534, in _on_message
self._on_message_callback(message)
File "C:\Users\mayan\anaconda3\lib\site-packages\jupyter_server_proxy\handlers.py", line 298, in message_cb
self.write_message(message, binary=isinstance(message, bytes))
File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\websocket.py", line 339, in write_message
raise WebSocketClosedError()
tornado.websocket.WebSocketClosedError
So what's the way forward on this? It seems the conclusion from #181 is that a whole new implementation is needed?
This issue has been mentioned by Mathworks concerning the official MATLAB kernel for Windows: https://blogs.mathworks.com/matlab/2023/01/30/official-mathworks-matlab-kernel-for-jupyter-released/
Something based on anyio
is probably the right path forward. We've handled some of these things on https://github.com/jupyter-lsp/jupyterlab-lsp, but it's far from optimal. But this is definitely something someone on a windows box would probably have to pick up. If mathworks wants to help out, that would be swell.
So if last week's PR (@rashedmyt) for simpervisor is accepted, do we expect this to be solved with the next simpervisor release?
Support for windows added via
- #392
- https://github.com/jupyterhub/simpervisor/pull/26