panel
panel copied to clipboard
Performance problems with panel > 0.13 and Lab3
We are experiencing some performance behaviors with panel > 0.13 The problems we are facing are:
- General lab slowness and input lags on Jupyter Lab
- Random kernel crashes while using tools (can also happen in non Panel tools)
- On local Windows machined we cannot use any panel related imports in startup.py files, using panel in said files will cause the kernel to crash and never boot.
We get the following error:
Traceback (most recent call last):
File "/opt/conda/bin/jupyterhub-singleuser", line 10, in <module>
sys.exit(main())
File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/app.py", line 66, in main
return SingleUserLabApp.launch_instance()
File "/opt/conda/lib/python3.9/site-packages/jupyter_core/application.py", line 264, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance
app.start()
File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/mixins.py", line 645, in start
super().start()
File "/opt/conda/lib/python3.9/site-packages/jupyter_server/serverapp.py", line 2669, in start
self.start_ioloop()
File "/opt/conda/lib/python3.9/site-packages/jupyter_server/serverapp.py", line 2655, in start_ioloop
self.io_loop.start()
File "/opt/conda/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start
self.asyncio_loop.run_forever()
File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
self._run_once()
File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 1875, in _run_once
handle = self._ready.popleft()
IndexError: pop from an empty deque
Here is the boken system pip list:
Package Version Editable project location
----------------------------- --------------------------- --------------------------------------------------------------------
aenum 2.2.4
aiohttp 3.8.1
aiosignal 1.2.0
alabaster 0.7.12
altair 4.1.0
analytics-python 1.4.0
anyio 3.6.1
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
asttokens 2.0.5
async-timeout 4.0.2
atomicwrites 1.4.0
attrs 21.4.0
awscli 1.23.9
Babel 2.10.1
backcall 0.2.0
backoff 1.10.0
beautifulsoup4 4.11.1
bleach 5.0.0
bokeh 2.4.3
boto3 1.22.9
botocore 1.25.9
cachetools 5.2.0
certifi 2022.5.18.1
cffi 1.15.0
charset-normalizer 2.0.12
cloudpickle 2.1.0
colorama 0.4.4
coverage 6.4.1
cycler 0.11.0
debugpy 1.6.0
decorator 5.1.1
defusedxml 0.7.1
docopt 0.6.2
docutils 0.15.2
entrypoints 0.4
execnet 1.9.0
executing 0.8.3
fastjsonschema 2.15.3
filelock 3.3.1
flake8 3.8.3
fonttools 4.33.3
frozenlist 1.3.0
fsspec 2022.5.0
gcsfs 2022.5.0
google-api-core 2.8.1
google-auth 2.6.6
google-auth-oauthlib 0.5.1
google-cloud-core 2.3.0
google-cloud-storage 2.3.0
google-crc32c 1.3.0
google-resumable-media 2.3.3
googleapis-common-protos 1.56.2
hybridcontents 0.3.0
icdiff 2.0.5
idna 3.3
imagesize 1.3.0
importlib-metadata 4.11.4
imubit-analytics 0.0.3
iniconfig 1.1.1
iniherit 0.3.9
ipyaggrid 0.2.1
ipydatetime 1.2.0
ipykernel 6.13.1
ipython 8.4.0
ipython-genutils 0.2.0
ipywidgets 7.7.0
isort 5.6.4
jedi 0.18.1
jenkinsapi 0.3.11
Jinja2 3.1.2
jmespath 1.0.0
joblib 1.1.0
json5 0.9.8
jsonschema 4.6.0
jupyter 1.0.0
jupyter-bokeh 3.0.4
jupyter-client 7.3.3
jupyter-console 6.4.3
jupyter-core 4.10.0
jupyter-server 1.17.1
jupyterlab 3.4.2
jupyterlab-pygments 0.2.2
jupyterlab-server 2.14.0
jupyterlab-widgets 1.1.0
kiwisolver 1.4.2
llvmlite 0.38.1
Markdown 3.3.7
MarkupSafe 2.1.1
matplotlib 3.5.1
matplotlib-inline 0.1.3
mccabe 0.6.1
mistune 0.8.4
mock 3.0.5
mock-generator 1.0.0
monotonic 1.6
multidict 6.0.2
nbclassic 0.3.7
nbclient 0.6.4
nbconvert 6.5.0
nbformat 5.4.0
nbval 0.9.6
nest-asyncio 1.5.5
nose 1.3.7
notebook 6.4.12
notebook-shim 0.1.0
numba 0.55.2
numexpr 2.8.1
numpy 1.22.0
oauthlib 3.2.0
packaging 21.3
pandas 1.2.5
pandocfilters 1.5.0
panel 0.13.1
param 1.12.1
parso 0.8.3
pickleshare 0.7.5
Pillow 9.1.1
pip 22.1.2
pluggy 0.13.1
pprintpp 0.4.0
prometheus-client 0.14.1
prompt-toolkit 3.0.29
protobuf 3.20.1
psutil 5.9.1
psycopg2-binary 2.9.3
pure-eval 0.2.2
py 1.11.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycodestyle 2.6.0
pycparser 2.21
pyct 0.4.8
pydantic 1.9.0
pyflakes 2.2.0
Pygments 2.12.0
pymongo 4.0.1
pyparsing 3.0.9
PyPika 0.48.8
pyrsistent 0.18.1
pytest 6.2.3
pytest-asyncio 0.14.0
pytest-cov 2.11.1
pytest-forked 1.4.0
pytest-icdiff 0.5
pytest-mock 3.6.1
pytest-repeat 0.9.1
pytest-xdist 2.4.0
python-dateutil 2.8.2
python-json-logger 0.1.11
python-slugify 6.1.2
pytimeparse 1.1.8
pytz 2022.1
pyviz-comms 2.2.0
pywin32 304
pywinpty 2.0.5
PyYAML 5.3.1
pyzmq 23.1.0
qtconsole 5.3.1
QtPy 2.1.0
requests 2.27.1
requests-oauthlib 1.3.1
rsa 4.7.2
s3contents 0.6.0
s3fs 0.4.2
s3transfer 0.5.2
scikit-learn 1.1.1
scipy 1.8.1
seaborn 0.11.2
selenium 3.141.0
Send2Trash 1.8.0
setuptools 62.3.3
shap 0.40.0
simpleeval 0.9.11
simplejson 3.17.6
six 1.16.0
sklearn 0.0
slicer 0.0.7
sniffio 1.2.0
snowballstemmer 2.2.0
soupsieve 2.3.2.post1
Sphinx 3.2.1
sphinx-rtd-theme 0.5.0
sphinxcontrib-applehelp 1.0.2
sphinxcontrib-devhelp 1.0.2
sphinxcontrib-htmlhelp 2.0.0
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.3
sphinxcontrib-serializinghtml 1.1.5
stack-data 0.2.0
tables 3.7.0
tabulate 0.8.9
terminado 0.15.0
testpath 0.6.0
text-unidecode 1.3
threadpoolctl 3.1.0
timedb 3.29.2
tinycss2 1.1.1
toml 0.10.2
toolz 0.11.2
tornado 6.1
traitlets 5.2.2.post1
typing_extensions 4.2.0
urllib3 1.26.9
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 1.3.2
wheel 0.34.2
widgetsnbextension 3.6.0
yapf 0.30.0
yarl 1.7.2
yoyo-migrations 5.0.5
zipp 3.8.0
And here is a pip list of the working environment:
Package Version Editable project location
--------------------------------- --------------------------- ------------------------------------------------------------------
aenum 2.2.4
aiohttp 3.8.1
aiosignal 1.2.0
alabaster 0.7.12
altair 4.1.0
analytics-python 1.4.0
anyio 3.6.1
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
asttokens 2.0.5
async-timeout 4.0.2
atomicwrites 1.4.0
attrs 21.4.0
awscli 1.23.9
Babel 2.10.1
backcall 0.2.0
backoff 1.10.0
beautifulsoup4 4.11.1
bleach 5.0.0
bokeh 2.4.3
boto3 1.22.9
botocore 1.25.9
cachetools 5.2.0
certifi 2022.5.18.1
cffi 1.15.0
charset-normalizer 2.0.12
click 8.0.3
cloudpickle 2.1.0
colorama 0.4.4
coverage 6.4.1
cycler 0.11.0
debugpy 1.6.0
decorator 5.1.1
defusedxml 0.7.1
docopt 0.6.2
docutils 0.15.2
entrypoints 0.4
execnet 1.9.0
executing 0.8.3
Faker 13.3.4
fastjsonschema 2.15.3
filelock 3.3.1
flake8 3.8.3
fonttools 4.33.3
frozenlist 1.3.0
fsspec 2022.5.0
gcsfs 2022.5.0
google-api-core 2.8.1
google-auth 2.7.0
google-auth-oauthlib 0.5.2
google-cloud-core 2.3.1
google-cloud-storage 2.4.0
google-crc32c 1.3.0
google-resumable-media 2.3.3
googleapis-common-protos 1.56.2
hybridcontents 0.3.0
icdiff 2.0.5
idna 3.3
imagesize 1.3.0
importlib-metadata 4.11.4
imubit-analytics 0.0.3
iniconfig 1.1.1
iniherit 0.3.9
ipyaggrid 0.2.1
ipydatetime 1.2.0
ipykernel 6.13.1
ipython 8.4.0
ipython-genutils 0.2.0
ipywidgets 7.7.0
isort 5.6.4
jedi 0.18.1
jenkinsapi 0.3.11
Jinja2 3.1.2
jmespath 1.0.0
joblib 1.1.0
json5 0.9.8
jsonschema 4.6.0
jupyter-bokeh 3.0.4
jupyter-client 7.3.4
jupyter-contrib-core 0.3.3
jupyter-contrib-nbextensions 0.5.1
jupyter-core 4.10.0
jupyter-highlight-selected-word 0.2.0
jupyter-latex-envs 1.4.6
jupyter-nbextensions-configurator 0.4.1
jupyter-server 1.17.1
jupyterlab 3.4.0
jupyterlab-pygments 0.2.2
jupyterlab-server 2.14.0
jupyterlab-widgets 1.1.0
jupyterthemes 0.20.0
kiwisolver 1.4.3
kwargs 1.0.1
lesscpy 0.15.0
llvmlite 0.38.1
lxml 4.7.1
Markdown 3.3.7
MarkupSafe 2.1.1
matplotlib 3.5.1
matplotlib-inline 0.1.3
mccabe 0.6.1
mistune 0.8.4
mock 3.0.5
mock-generator 1.0.0
monotonic 1.6
multidict 6.0.2
mypy-extensions 0.4.3
nbclassic 0.3.7
nbclient 0.6.4
nbconvert 6.5.0
nbformat 5.4.0
nbval 0.9.6
nest-asyncio 1.5.5
nose 1.3.7
notebook 6.4.12
notebook-shim 0.1.0
numba 0.55.2
numexpr 2.8.1
numpy 1.22.4
oauthlib 3.2.0
packaging 21.3
pandas 1.2.5
pandocfilters 1.5.0
panel 0.13.1
param 1.12.1
parameterized 0.8.1
parso 0.8.3
pathspec 0.9.0
pbr 5.8.1
pickleshare 0.7.5
Pillow 9.1.1
pip 22.1.2
platformdirs 2.4.1
pluggy 0.13.1
ply 3.11
pprintpp 0.4.0
progress 1.6
prometheus-client 0.14.1
prompt-toolkit 3.0.29
protobuf 3.20.1
psutil 5.9.1
psycopg2-binary 2.9.3
pure-eval 0.2.2
py 1.11.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycodestyle 2.6.0
pycparser 2.21
pyct 0.4.8
pydantic 1.9.0
pyflakes 2.2.0
Pygments 2.12.0
pymongo 4.0.1
pyparsing 3.0.9
PyPika 0.48.8
pyrsistent 0.18.1
PyRSS2Gen 1.1
pytest 6.2.3
pytest-asyncio 0.14.0
pytest-cov 2.11.1
pytest-datadir-ng 1.1.1
pytest-forked 1.4.0
pytest-icdiff 0.5
pytest-mock 3.6.1
pytest-pythonpath 0.7.4
pytest-repeat 0.9.1
pytest-xdist 2.4.0
python-dateutil 2.8.2
python-json-logger 0.1.11
python-slugify 6.1.2
pytimeparse 1.1.8
pytz 2022.1
pyviz-comms 2.2.0
pywin32 304
pywinpty 2.0.5
PyYAML 6.0
pyzmq 23.1.0
requests 2.28.0
requests-oauthlib 1.3.1
rsa 4.7.2
s3contents 0.6.0
s3fs 0.4.2
s3transfer 0.5.2
scikit-learn 1.1.1
scipy 1.8.1
seaborn 0.11.2
selenium 3.141.0
self 2020.12.3
Send2Trash 1.8.0
setuptools 62.4.0
shap 0.40.0
simpleeval 0.9.11
simplejson 3.17.6
six 1.16.0
sklearn 0.0
slicer 0.0.7
sniffio 1.2.0
snowballstemmer 2.2.0
soupsieve 2.3.2.post1
Sphinx 3.2.1
sphinx-rtd-theme 0.5.0
sphinxcontrib-applehelp 1.0.2
sphinxcontrib-devhelp 1.0.2
sphinxcontrib-htmlhelp 2.0.0
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.3
sphinxcontrib-serializinghtml 1.1.5
sqlparse 0.4.2
stack-data 0.2.0
success 1.0.0
tables 3.7.0
tabulate 0.8.9
terminado 0.15.0
testpath 0.6.0
text-unidecode 1.3
threadpoolctl 3.1.0
timedb 3.29.2
tinycss2 1.1.1
toml 0.10.2
tomli 1.2.3
toolz 0.11.2
tornado 6.1
tqdm 4.64.0
traitlets 5.2.2.post1
typing_extensions 4.2.0
urllib3 1.26.9
versioneer 0.22
virtualenv 20.14.1
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 1.3.2
wheel 0.34.2
widgetsnbextension 3.6.0
xeus-python 0.13.6
xeus-python-shell 0.2.0
yapf 0.30.0
yarl 1.7.2
yoyo-migrations 5.0.5
zipp 3.8.0
I was aware of performance regressions in 0.13.0 (see https://github.com/holoviz/panel/issues/3531) but was under the impression these were fixed. You are certain you have updated to 0.13.1 in your base env (the one that the Jupyter server is running in)?
Hey @philippjfr, I'm working with @ItamarShDev, we did tried 0.13.1 and even 0.14.0a1 and 0.14.0a2, but the problems listed above consisted.
Thanks for the update, someone will have to do a bisection on the codebase. Can you give me a very simple, step-wise recipe that will reproduce the issue so that bisection doesn't take forever?
The first two sections written above are hard to reproduce
The first one (the general slowness) is always there from the moment we starting Jupyter (with panel imported trough the startup folder).
The second one (the kernel crashes) are happening suddenly during work, and the exception mentioned above is shown in the logs.
The third one (kernel wont start / crash) I can reproduce easily in windows, I'm creating a file with just line for importing panel and save it under the startup folder (%user_home%\.ipython\profile_default\startup
).
We posted the list of packages above we are using above, we are using jupyterlab
version 3.4.0
and we tried panel
version 0.13.*
and above.
just to make it clearer when go back to older panel (0.12.7 and below) stuff are smoother, the kernel exception didn't happened during my test time and the import from windows using the startup folder worked smoothly and the kernel started without any issue.
LMK if you need any other information.
thanks.
Thanks, that's very helpful!
Hey @philippjfr, do you have any news for us ?
If any tess are needed, we will be happy to test
Sorry, don't have any updates yet. Been a little swamped. Hope to have some time tomorrow.
Thanks for the response @philippjfr, looking forward for it.
I also experience this when working on my windows laptop on the current Panel master branch.
Sorry I haven't gotten to this. Unfortunately I've been travelling almost non stop for weeks and havent had access to a windows machine.
On this amazing youtube video by Thu Vue she has to state around minute 04:00 that you should not install Panel in the same virtual environment as Jupyterlab due to some weird issue https://www.youtube.com/watch?v=B11nK1QCPRA.
I believe this is most important bug to fix for Panel as you cannot currently work with the latest version of Panel and Jupyter lab on Windows if you are not an experienced user willing to put in some effort to get it working.
Would be happy to backport this to a 0.13.x release asap if we figure out what is causing it. If someone with a windows machine could do a git bisect that would be tremendously helpful. I won't be able to do it until Sunday at the very earliest.
@philippjfr If you could guide me through, i can give you any Windows related info.
So sorry this has taken so long but I was finally able to test this on windows and I have had absolutely no luck reproducing the issue. Maybe we can sync at some point and debug together.
@maximlt or @Hoxbro do either of you have a windows machine where you could try to replicate?
By using the first environment on Windows and adding import panel as pn
to a startup file, I get the same no response when running in Jupyter Lab. If I add another check to these lines 'jupyter_client' not in sys.modules
, it seems to work as expected.
https://github.com/holoviz/panel/blob/8c5ca7a1e16c84f781ee137d3f709fa2370e74d5/panel/io/server.py#L538-L545
I could not see any slowness except that. The log message in the first comment also seems to indicate it running on Linux and not Windows.
Wow, great sleuthing thanks! Could you submit a PR adding that additional guard?
Note that the IndexError: pop from an empty deque
is unrelated to Panel and caused by some changes in tornado>=6.1:
https://github.com/jupyterlab/jupyterlab/issues/11934