jupyter-server-proxy

jupyter-server-proxy copied to clipboard

When trying to run jupyterlab-nvdashboard on windows I'm getting the following error on startup.

(Sorry about the screenshot, copying from powershell seems to lose all linebreaks)

I imagine this is related to the way the server proxy is using asyncio/tornado to create subprocesses.

Conda env

name: pipeline
channels:
  - conda-forge
  - defaults
dependencies:
  - attrs=19.1.0=py_0
  - backcall=0.1.0=py_0
  - bleach=3.1.0=py_0
  - bokeh=1.3.4=py37_0
  - ca-certificates=2019.5.15=1
  - certifi=2019.6.16=py37_1
  - click=7.0=py_0
  - cloudpickle=1.2.2=py_0
  - colorama=0.4.1=py_0
  - cudatoolkit=10.0.130=0
  - cudnn=7.6.0=cuda10.0_0
  - cupy=6.0.0=py37h230ac6f_0
  - cytoolz=0.10.0=py37hfa6e2cd_0
  - dask=2.4.0=py_0
  - dask-core=2.4.0=py_0
  - decorator=4.4.0=py_0
  - defusedxml=0.5.0=py_1
  - distributed=2.4.0=py_0
  - entrypoints=0.3=py37_1000
  - fastrlock=0.4=py37h6538335_1000
  - freetype=2.10.0=h563cfd7_1
  - fsspec=0.5.1=py_0
  - heapdict=1.0.0=py37_1000
  - intel-openmp=2019.5=281
  - ipykernel=5.1.2=py37h5ca1d4c_0
  - ipython=7.8.0=py37h5ca1d4c_0
  - ipython_genutils=0.2.0=py_1
  - jedi=0.15.1=py37_0
  - jinja2=2.10.1=py_0
  - jpeg=9c=hfa6e2cd_1001
  - json5=0.8.5=py_0
  - jsonschema=3.0.2=py37_0
  - jupyter_client=5.3.1=py_0
  - jupyter_core=4.4.0=py_0
  - jupyterlab=1.1.4=py_0
  - jupyterlab_server=1.0.6=py_0
  - libblas=3.8.0=12_mkl
  - libcblas=3.8.0=12_mkl
  - liblapack=3.8.0=12_mkl
  - libpng=1.6.37=h7602738_0
  - libsodium=1.0.17=h2fa13f4_0
  - libtiff=4.0.10=h6512ee2_1003
  - llvmlite=0.29.0=py37hed17590_1
  - locket=0.2.0=py_2
  - lz4-c=1.8.3=he025d50_1001
  - m2w64-gcc-libgfortran=5.3.0=6
  - m2w64-gcc-libs=5.3.0=7
  - m2w64-gcc-libs-core=5.3.0=7
  - m2w64-gmp=6.1.0=2
  - m2w64-libwinpthread-git=5.0.0.4634.697f757=2
  - markupsafe=1.1.1=py37hfa6e2cd_0
  - mistune=0.8.4=py37hfa6e2cd_1000
  - mkl=2019.4=245
  - msgpack-python=0.6.2=py37he980bc4_0
  - msys2-conda-epoch=20160418=1
  - nbconvert=5.6.0=py37_1
  - nbformat=4.4.0=py_1
  - nodejs=10.13.0=0
  - notebook=6.0.1=py37_0
  - numba=0.45.1=py37hf9181ef_0
  - numpy=1.15.4=py37h8078771_1002
  - olefile=0.46=py_0
  - openssl=1.1.1d=he774522_0
  - packaging=19.2=py_0
  - pandas=0.25.1=py37he350917_0
  - pandoc=2.7.3=0
  - pandocfilters=1.4.2=py_1
  - parso=0.5.1=py_0
  - partd=1.0.0=py_0
  - pickleshare=0.7.5=py37_1000
  - pillow=6.1.0=py37h643dfcc_1
  - pip=19.2.3=py37_0
  - prometheus_client=0.7.1=py_0
  - prompt_toolkit=2.0.9=py_0
  - psutil=5.6.3=py37hfa6e2cd_0
  - pygments=2.4.2=py_0
  - pyparsing=2.4.2=py_0
  - pyrsistent=0.15.4=py37hfa6e2cd_0
  - python=3.7.3=h510b542_1
  - python-dateutil=2.8.0=py_0
  - pytz=2019.2=py_0
  - pywinpty=0.5.5=py37_1000
  - pyyaml=5.1.2=py37hfa6e2cd_0
  - pyzmq=18.1.0=py37h16f9016_0
  - send2trash=1.5.0=py_0
  - setuptools=41.2.0=py37_0
  - six=1.12.0=py37_1000
  - sortedcontainers=2.1.0=py_0
  - sqlite=3.29.0=hfa6e2cd_1
  - tbb=2019.8=he980bc4_0
  - tblib=1.4.0=py_0
  - terminado=0.8.2=py37_0
  - testpath=0.4.2=py_1001
  - tk=8.6.9=hfa6e2cd_1003
  - toolz=0.10.0=py_0
  - tornado=6.0.3=py37hfa6e2cd_0
  - traitlets=4.3.2=py37_1000
  - vc=14.1=h0510ff6_4
  - vs2015_runtime=14.16.27012=hf0eaf9b_0
  - wcwidth=0.1.7=py_1
  - webencodings=0.5.1=py_1
  - wheel=0.33.6=py37_0
  - wincertstore=0.2=py37_1002
  - winpty=0.4.3=4
  - xz=5.2.4=h2fa13f4_1001
  - yaml=0.1.7=hfa6e2cd_1001
  - zeromq=4.3.2=h6538335_2
  - zict=1.0.0=py_0
  - zlib=1.2.11=h2fa13f4_1006
  - zstd=1.4.0=hd8a0e53_0
  - pip:
    - aiohttp==3.6.1
    - async-timeout==3.0.1
    - chardet==3.0.4
    - idna==2.8
    - jupyter-server-proxy==1.1.0
    - jupyterlab-nvdashboard==0.1.9
    - multidict==4.5.2
    - pynvml==8.0.3
    - simpervisor==0.3
    - yarl==1.3.0
prefix: C:\Users\jacob\Miniconda3\envs\pipeline

Thanks for the report @jacobtomlinson. Yes, it seems that some asyncio features used by simpervisor are not supported on Windows. I don't know if this can be addressed within simpervisor -- that's out of my depth. I think the most we can do here is document this issue.

I came across the same issue on windows, and made an issue in the simpervisor repo.

I did a bit more research on the windows support, my conclusion is that we may need to implement a fallback solution for windows which does not use simpervisor.

The reason is:

The default asyncio event loop implementation on Windows does not support subprocesses. Subprocesses are available for Windows if a ProactorEventLoop is used. See Subprocess Support on Windows for details.

https://docs.python.org/3/library/asyncio-subprocess.html

I tried to switch the eventloop for simpervisor:

if sys.platform == 'win32':
    loop = asyncio.ProactorEventLoop()
    asyncio.set_event_loop(loop)

However, the entire jupyter server hangs, and I figured it maybe because tornado server doesn't not support ProactorEventLoop:

https://github.com/tornadoweb/tornado/blob/d6819307ee050bbd8ec5deb623e9150ce2220ef9/tornado/platform/asyncio.py#L15-L19

We may need to implement a synchronous version of the SupervisedProcess, and run it in a thread/executor instead.

@jacobtomlinson I made a PR for this issue, would be great if you can do a test by running pip install -U http://github.com/oeway/jupyter-server-proxy/tarball/master#egg=jupyter-server-proxy?

@oeway Tried this, does not work with nvdashboard.

After pip install -U http://github.com/oeway/jupyter-server-proxy/tarball/master#egg=jupyter-server-proxy , I even build jupyter labextension install jupyterlab-nvdashboard

Here is the stacktrace

Traceback (most recent call last):
  File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\web.py", line 1703, in _execute
    result = await result
  File "C:\Users\mayan\anaconda3\lib\site-packages\jupyter_server_proxy\websocket.py", line 97, in get
    return await self.http_get(*args, **kwargs)
  File "C:\Users\mayan\anaconda3\lib\site-packages\jupyter_server_proxy\handlers.py", line 534, in http_get
    return await self.proxy(self.port, path)
  File "C:\Users\mayan\anaconda3\lib\site-packages\jupyter_server_proxy\handlers.py", line 528, in proxy
    await self.ensure_process()
  File "C:\Users\mayan\anaconda3\lib\site-packages\jupyter_server_proxy\handlers.py", line 505, in ensure_process
    await proc.start()
  File "C:\Users\mayan\anaconda3\lib\site-packages\simpervisor\process.py", line 90, in start
    self.proc = await asyncio.create_subprocess_exec(
  File "C:\Users\mayan\anaconda3\lib\asyncio\subprocess.py", line 236, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "C:\Users\mayan\anaconda3\lib\asyncio\base_events.py", line 1630, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "C:\Users\mayan\anaconda3\lib\asyncio\base_events.py", line 491, in _make_subprocess_transport
    raise NotImplementedError
NotImplementedError

@oeway My bad, it works!

After pip install -U http://github.com/oeway/jupyter-server-proxy/tarball/master#egg=jupyter-server-proxy

Just run jupyter lab via the terminal, no need to build anything. The ProactorEventLoop takes care of it. 💯

There are a couple of uncaught exceptions thrown by the tornado web-server which I am listing here:

[E 17:57:37.908 LabApp] Uncaught exception
    Traceback (most recent call last):
      File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\websocket.py", line 649, in _run_callback
        result = callback(*args, **kwargs)
      File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\websocket.py", line 1528, in on_message
        return self._on_message(message)
      File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\websocket.py", line 1534, in _on_message
        self._on_message_callback(message)
      File "C:\Users\mayan\anaconda3\lib\site-packages\jupyter_server_proxy\handlers.py", line 298, in message_cb
        self.write_message(message, binary=isinstance(message, bytes))
      File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\websocket.py", line 339, in write_message
        raise WebSocketClosedError()
    tornado.websocket.WebSocketClosedError
[E 17:57:37.990 LabApp] Uncaught exception
    Traceback (most recent call last):
      File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\websocket.py", line 649, in _run_callback
        result = callback(*args, **kwargs)
      File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\websocket.py", line 1528, in on_message
        return self._on_message(message)
      File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\websocket.py", line 1534, in _on_message
        self._on_message_callback(message)
      File "C:\Users\mayan\anaconda3\lib\site-packages\jupyter_server_proxy\handlers.py", line 298, in message_cb
        self.write_message(message, binary=isinstance(message, bytes))
      File "C:\Users\mayan\anaconda3\lib\site-packages\tornado\websocket.py", line 339, in write_message
        raise WebSocketClosedError()
    tornado.websocket.WebSocketClosedError

So what's the way forward on this? It seems the conclusion from #181 is that a whole new implementation is needed?

This issue has been mentioned by Mathworks concerning the official MATLAB kernel for Windows: https://blogs.mathworks.com/matlab/2023/01/30/official-mathworks-matlab-kernel-for-jupyter-released/

Something based on anyio is probably the right path forward. We've handled some of these things on https://github.com/jupyter-lsp/jupyterlab-lsp, but it's far from optimal. But this is definitely something someone on a windows box would probably have to pick up. If mathworks wants to help out, that would be swell.

So if last week's PR (@rashedmyt) for simpervisor is accepted, do we expect this to be solved with the next simpervisor release?

Support for windows added via

• #392
• https://github.com/jupyterhub/simpervisor/pull/26