ipywidgets
ipywidgets copied to clipboard
jupyter-widgets
CONTROL_COMM_TIMEOUT too low to work for some notebooks
Description
We are running some rather complicated notebooks using voila. They work fine when testing locally but when deployed to a remote server they stop working, and we see the following message in the browser log:
Failed to fetch ipywidgets through the "jupyter.widget.control" comm channel, fallback to fetching individual model state. Reason: Control comm did not respond in time
I eventually traced this down to the value of CONTROL_COMM_TIMEOUT, by manually changing this to a much higher value the notebooks work as expected. There doesn't appear to be a way to do this from configuration however.
Reproduce
Start a complicated notebook over an internet connection.
I set a breakpoint on the error being printed and worked my way backwards. Manually changed the 4e3 to 4e8 in the compiled javascript and the problem went away
Expected behavior
I expect the control channel to be retried, perhaps an exponential backoff to a larger value, or ideally a way to just increase the value as a site specific setting.
Context
- ipywidgets version: 8.0.2
- Operating System and version: Fedora 36
- Browser and version: Firefox 106.0.1
Troubleshoot Output
# jupyter troubleshoot $PATH: /root/.local/bin /root/bin /usr/local/sbin /usr/local/bin /usr/sbin /usr/bin /sbin /binsys.path: /usr/local/bin /usr/lib64/python310.zip /usr/lib64/python3.10 /usr/lib64/python3.10/lib-dynload /usr/local/lib64/python3.10/site-packages /usr/local/lib/python3.10/site-packages /usr/lib64/python3.10/site-packages /usr/lib/python3.10/site-packages
sys.executable: /usr/bin/python3
sys.version: 3.10.8 (main, Oct 12 2022, 00:00:00) [GCC 12.2.1 20220819 (Red Hat 12.2.1-2)]
platform.platform(): Linux-5.17.11-300.fc36.x86_64-x86_64-with-glibc2.35
pip list: Package Version --------------------- --------- anyio 3.6.2 appdirs 1.4.4 argon2-cffi 21.3.0 argon2-cffi-bindings 21.2.0 asttokens 2.1.0 attrs 21.4.0 Babel 2.11.0 backcall 0.2.0 beautifulsoup4 4.11.0 bleach 5.0.1 boto3 1.21.33 botocore 1.24.33 Bottleneck 1.3.2 branca 0.5.0 Brotli 1.0.9 cachetools 5.0.0 certifi 2021.10.8 cffi 1.15.0 charset-normalizer 2.0.11 click 8.0.4 click-plugins 1.1.1 cligj 0.7.2 cryptography 36.0.0 cycler 0.11.0 dbus-python 1.2.18 debugpy 1.6.3 decorator 5.1.1 defusedxml 0.7.1 entrypoints 0.4 et-xmlfile 1.0.1 executing 1.2.0 fastjsonschema 2.16.2 Fiona 1.8.21 folium 0.13.0 fonttools 4.29.1 fs 2.4.11 fsspec 2022.10.0 gcsfs 0.6.2 geopandas 0.11.1 google-auth 2.14.0 google-auth-oauthlib 0.4.1 gpg 1.17.0 greenlet 1.1.2 html5lib 1.1 htmlmin 0.1.12 idna 3.3 ImageHash 4.3.1 ipykernel 6.17.1 ipyleaflet 0.17.2 ipython 8.6.0 ipython-genutils 0.2.0 ipywidgets 8.0.2 jdcal 1.4.1 jedi 0.18.1 Jinja2 3.0.3 jmespath 1.0.0 joblib 1.2.0 json5 0.9.10 jsonschema 4.17.0 jupyter_client 7.4.1 jupyter_core 5.0.0 jupyter-server 1.23.2 jupyterlab-pygments 0.2.2 jupyterlab_server 2.16.3 jupyterlab-widgets 3.0.3 kiwisolver 1.4.1 libcomps 0.1.18 lxml 4.7.1 MarkupSafe 2.1.1 matplotlib 3.5.3 matplotlib-inline 0.1.6 missingno 0.5.1 mistune 2.0.4 multimethod 1.9 munch 2.5.0 munkres 1.1.2 nbclassic 0.4.8 nbclient 0.7.0 nbconvert 7.2.5 nbformat 5.7.0 nest-asyncio 1.5.6 networkx 2.8.8 notebook 6.5.2 notebook_shim 0.2.2 numexpr 2.8.1 numpy 1.22.0 oauthlib 3.0.2 odfpy 1.4.1 olefile 0.46 openpyxl 3.0.3 packaging 21.3 pandas 1.3.5 pandas-datareader 0.10.0 pandas-profiling 3.4.0 pandocfilters 1.5.0 parso 0.8.3 patsy 0.5.3 pexpect 4.8.0 phik 0.12.2 pickleshare 0.7.5 Pillow 9.1.0 pip 21.3.1 platformdirs 2.5.4 ply 3.11 prometheus-client 0.15.0 prompt-toolkit 3.0.32 psutil 5.9.4 psycopg2 2.9.1 ptyprocess 0.7.0 pure-eval 0.2.2 pyasn1 0.4.8 pyasn1-modules 0.2.8 pycairo 1.21.0 pycparser 2.20 pydantic 1.10.2 Pygments 2.13.0 PyJWT 2.4.0 PyMySQL 0.10.1 pyparsing 2.4.7 pyproj 3.4.0 PyQt5 5.15.6 PyQt5-sip 12.9.0 pyrsistent 0.19.2 PySocks 1.7.1 python-dateutil 2.8.2 pytz 2022.6 PyWavelets 1.4.1 PyYAML 6.0 pyzmq 24.0.1 QtPy 2.0.0 requests 2.27.1 requests-oauthlib 1.3.0 rpm 4.17.1 rsa 4.8 s3transfer 0.5.1 scipy 1.8.1 seaborn 0.12.1 Send2Trash 1.8.0 setuptools 59.6.0 Shapely 1.8.5 six 1.16.0 sniffio 1.3.0 soupsieve 2.3.1 SQLAlchemy 1.4.42 stack-data 0.6.1 statsmodels 0.13.5 tables 3.7.0 tabulate 0.8.10 tangled-up-in-unicode 0.2.0 terminado 0.17.0 tinycss2 1.2.1 tornado 6.2 tqdm 4.64.1 traitlets 5.5.0 traittypes 0.2.1 typing_extensions 4.4.0 urllib3 1.26.12 visions 0.7.5 voila 0.4.0 wcwidth 0.2.5 webencodings 0.5.1 websocket-client 1.4.2 websockets 10.4 widgetsnbextension 4.0.3 xarray 2022.6.0 xeus-python 0.14.3 xeus-python-shell 0.5.0 xlrd 2.0.1 XlsxWriter 3.0.3 xlwt 1.3.0 xyzservices 2022.9.0
Command Line Output
N/A
Browser Output
Failed to fetch ipywidgets through the "jupyter.widget.control" comm channel, fallback to fetching individual model state. Reason: Control comm did not respond in time
If using JupyterLab
- JupyterLab version:
Installed Labextensions
N/A
This is related to https://github.com/jupyter-widgets/ipywidgets/issues/3580
From our dev discussion in our dev meeting today:
- We should increase the timeout.
- We can look into an early reply from the control comm.
- We should make sure that when we abort this control comm, that we really do abort the codepath.
Thank you! For now my local workaround works (a wonderful sed hack in the Dockerfile) but a proper fix would be very nice!
Thank you! For now my local workaround works (a wonderful sed hack in the Dockerfile) but a proper fix would be very nice!
@hpvb, just wondering how you've hacked this for now as I am having the exact issue. How exactly are you editing the CONTROL_COMM_TIMEOUT value?
@jasongrout - sounds encouraging.
For some extra context, @ollyhensby and I are working on quite a large / complex application. It works fine in JupyterLab but we get this issue in Voila. To make matters worse - it only happens sometimes! Most of the time the app works fine, but on occasion it is blank on loading (i guess depending on usage). Would be great to see a fix.
Do you also see "Comm is already created" as I mention in https://github.com/jupyter-widgets/ipywidgets/issues/3580#issue-1357330645 ?
Just to add @maartenbreddels, we've got around this issue for now by removing initial loading of some widgets and instead have them load when observed e.g. we have tabs where when a tab is clicked on, the widgets within it load upon the tab being selected.
Doing this we therefore load a lot less widgets upon initialising and avoid the error shown above.
This is voila 0.4.0, so that means from https://github.com/voila-dashboards/voila/blob/0.4.x/packages/voila/package.json
We are using https://github.com/jupyter-widgets/ipywidgets/blame/511663a56324cea5324f49a65ebe25e2f1b04d87/packages/base-manager/src/manager-base.ts#L382 which hasn't change for 2 years, so that is likely the code that you are running.
I think there is an issue in the protocol as it now is (I discussed this with @mariobuikhuizen a while ago when analyzing this because of a different bug). IIRC, we suggested back then (private communication) that we should change the protocol such that when the comm is opened, the kernel sends back an 'acknowledge' message first, after which we should ignore the timeout. Now we do not know if the kernel supports the control comm or not, and let the timeout logic assume that. After the acknowledge, we should just wait for the data.
@martinRenou IIRC there is a proposal somewhere to put this in the Comm protocol, is that right?
In Solara server we take a different approach, where we only start running the Python code when the client is connected, avoiding this issue.
Yes, that is a good workaround and idea in general. You might be interested in Reacton, which will automatically create and destroy widgets for you in such a situation. It does require investing a bit into learning to think like React (in Python)
The exact console warning that OP reports shows up for me when creating a new notebook in a new environment with only jupyterlab, jupyterlab-widgets, and their dependencies installed. No idea if the same cause, but it doesn't take a complicated notebook to get the warning.
Versions
Chrome 120.0.6099.234 anyio 4.2.0 argon2-cffi 23.1.0 argon2-cffi-bindings 21.2.0 arrow 1.3.0 asttokens 2.4.1 async-lru 2.0.4 attrs 23.2.0 Babel 2.14.0 backcall 0.2.0 beautifulsoup4 4.12.3 bleach 6.1.0 certifi 2023.11.17 cffi 1.16.0 charset-normalizer 3.3.2 comm 0.2.1 debugpy 1.8.0 decorator 5.1.1 defusedxml 0.7.1 exceptiongroup 1.2.0 executing 2.0.1 fastjsonschema 2.19.1 fqdn 1.5.1 idna 3.6 importlib-metadata 7.0.1 importlib-resources 6.1.1 ipykernel 6.29.0 ipython 8.12.3 isoduration 20.11.0 jedi 0.19.1 Jinja2 3.1.3 json5 0.9.14 jsonpointer 2.4 jsonschema 4.21.1 jsonschema-specifications 2023.12.1 jupyter_client 8.6.0 jupyter_core 5.7.1 jupyter-events 0.9.0 jupyter-lsp 2.2.2 jupyter_server 2.12.5 jupyter_server_terminals 0.5.1 jupyterlab 4.0.11 jupyterlab_pygments 0.3.0 jupyterlab_server 2.25.2 jupyterlab-widgets 3.0.9 MarkupSafe 2.1.3 matplotlib-inline 0.1.6 mistune 3.0.2 nbclient 0.9.0 nbconvert 7.14.2 nbformat 5.9.2 nest-asyncio 1.5.9 notebook_shim 0.2.3 overrides 7.4.0 packaging 23.2 pandocfilters 1.5.1 parso 0.8.3 pexpect 4.9.0 pickleshare 0.7.5 pip 23.3.2 pkgutil_resolve_name 1.3.10 platformdirs 4.1.0 prometheus-client 0.19.0 prompt-toolkit 3.0.43 psutil 5.9.8 ptyprocess 0.7.0 pure-eval 0.2.2 pycparser 2.21 Pygments 2.17.2 python-dateutil 2.8.2 python-json-logger 2.0.7 pytz 2023.3.post1 PyYAML 6.0.1 pyzmq 25.1.2 referencing 0.32.1 requests 2.31.0 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rpds-py 0.17.1 Send2Trash 1.8.2 setuptools 41.6.0 six 1.16.0 sniffio 1.3.0 soupsieve 2.5 stack-data 0.6.3 terminado 0.18.0 tinycss2 1.2.1 tomli 2.0.1 tornado 6.4 traitlets 5.14.1 types-python-dateutil 2.8.19.20240106 typing_extensions 4.9.0 uri-template 1.3.0 urllib3 2.1.0 wcwidth 0.2.13 webcolors 1.13 webencodings 0.5.1 websocket-client 1.7.0 zipp 3.17.0
