dask-labextension
dask-labextension copied to clipboard
dask
Automatically connect to new dashboard
A common workflow for me when using distributed is to layout my JupyterLab with a couple of dask panes so I can keep an eye on the cluster. I'm often starting new clusters and pasting new /status urls in the dask-labextension sidebar. When I paste in a new url the current dask panes show a 404 message and AFAICS the only way to get them to recognise the new connection is to close the existing panes, open new panes and then recreate the layout as I had it previously.
It would be great if the dask panes could recognise a new url and connect to it automatically. If there's a reason that can't work automatically, perhaps rather than a 404 message the panes could have a refresh button which would allow them to connect to a new url?
@dhirschfeld, thanks for the report. My intention was that the "search" icon next to the URL would attempt to automatically query the dashboard URL from the currently active notebook/console. Is it not working for your use-case? We may need to harden that a bit.
Or are the panes not correctly loading a valid URL? If so, can you provide a sequence of commands so I can try to reproduce (ideally with a local cluster)?
@ian-r-rose I'm having the same issue. Not sure what the search icon is doing, but it's not getting the correct ip. This is what happens after i restart the kernel and run everything up to the client part in cell [4]. It would be nice if this problem can get fixed, that way i can ''reload'' the current open windows, after restarting the kernel. Without having to re-open them and creating the layout i like to have.
TLDR: the search icon makes it get 'http://192.168.0.100/20840/1:8787/' while it should get 'http://localhost:8787/status'.
It looks like this is a problem within the dask/distributed library. The Cluster.dashboard_link property gets the host address of the scheduler, but if the scheduler is running using the inproc:// protocol rather than tcp:// then this gives a wacky address defined by the PID and port number.
This can be fixed here:
https://github.com/dask/distributed/blob/acc4b9076e034574533a960323e0d774b4f15e1e/distributed/deploy/cluster.py#L75-L80
Probably we want to replace host with localhost if self.scheduler.address starts with inproc://
@deKeijzer is this something that you have the time to contribute?
also cc @jacobtomlinson
@mrocklin Sadly not at the moment, but thanks for asking. I'm still quite busy with actually learning Dask haha.
Well then short term I recommend setting Client(processes=True) or just putting http://localhost:8787 into the search bar manually.
On Mon, Oct 1, 2018 at 11:27 AM Brian de Keijzer [email protected] wrote:
@mrocklin https://github.com/mrocklin Sadly not at the moment, but thanks for asking. I'm still quiet busy with actually learning Dask haha.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/dask-labextension/issues/26#issuecomment-425950069, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszNBkb8SQBhGaASh4mnXqJiG-ujzCks5ugjRQgaJpZM4W17u2 .
My intention was that the "search" icon next to the URL would attempt to automatically query the dashboard URL from the currently active notebook/console. Is it not working for your use-case? Or are the panes not correctly loading a valid URL?
What I'm observing is that when I close a dask cluster I see the 404: Not Found message in the open panes. When I then start a new cluster (using a different random port) and put in the correct url I observe the buttons all turn from grey to orange (so it has connected correctly) however the panes don't automatically refresh and recognise the new url - they continue to display the 404: Not Found message.
The only way to get the panes to recognise the new url is to close and then re-open them from the sidebar. This necessitates a lot of mucking about rearranging the layout you had previously so it would be great if the panes automatically refreshed the connection when you entered a new url.
This sounds like a different problem to what @deKeijzer is observing as my url is correct and the extension is connected (buttons are orange) - it's just the existing panes aren't recognising a new url.
@dhirschfeld I am having a tough time reproducing this. I have a notebook with a single cell:
from dask.distributed import Client
client = Client()
client
and am performing the following steps:
- Execute the cell.
- Click the "Search" icon, which correctly identifies the dashboard url.
- Open one of the dashboard panes.
- Restart the kernel, thus shutting down the cluster. The dashboard panel becomes empty.
- Execute the cell. A new client is started.
- Click the "Search" icon. The same panel I had open before automatically springs to life.
What browser are you using? I can reproduce this behavior in both firefox and chrome on Ubuntu.
The dashboard panel becomes empty
You mean you see the same 404: Not Found message?
I'm using chrome. My distributed scheduler is remote and every time I restart it gets mapped to a new port - that could be the key here.
I'll see if I can test further to narrow it down...
Any updates on this issue? I am experiencing similar greyed out buttons to access the dashboard and receive a connection refused when opening 'http://localhost:8888/proxy/8787/status' in a new tab in chrome.
Likewise, I use a PBS cluster and pangeo on cheyenne and port forward to see the lab notebook running remotely. I was once able to connect to a client and start some workers but also saw the 404: Not Found for the dask task stream.
Would love to get this up and running to figure out my worker load balance and see the task stream.
Chiming in here to say that I'm also having this problem, using dask-labextension on a slurm cluster behind a vpn. I can access the dashboard through an ssh tunnel, but putting the tunneled dashboard url (http://localhost:8787) into the search box does not appear to do anything.
@bmcfee are you also accessing Jupyter at localhost?
Just to check you can view the dashboard at http://localhost:8787 but if you put it into the box and press enter the buttons remain grey?
No, our jupyter setup runs a bit differently, and is not exposed as localhost.
Otherwise, that's correct: I can tunnel a port on localhost to the machine running the dask dashboard, visit the dashboard directly, but pasting the link doesn't work.
No, our jupyter setup runs a bit differently, and is not exposed as localhost.
This could be the problem then.
If you remove /lab from the end of your Jupyter URL and add /proxy/8787/status to the end are you able to view the dashboard?
If you remove
/labfrom the end of your Jupyter URL and add/proxy/8787/statusto the end are you able to view the dashboard?
Aha, that does seem to work!
Is there any way (eg from a cell magic or something) to push that over to labextension from a notebook cell? (Background: I'm using this for teaching purposes, and it'd be best if students didn't have to copy-mangle-paste to get the dashboard working.)
Actually, I spoke too soon -- it doesn't quite work. Modifying the url as you suggest does connect to the dashboard, and turns the links orange, but none of the dashboard contents show up.
However, if I connect to the dashboard through the ssh tunnel, everything does appear as expected. This might be down to a quirk of how our network is set up, in which case, feel free to ignore this. Seems strange though!
Is there any way (eg from a cell magic or something) to push that over to labextension from a notebook cell?
You can set the dashboard url templarte in your Dask config which is picked up by the Lab extension.
Modifying the url as you suggest does connect to the dashboard, and turns the links orange, but none of the dashboard contents show up.
Does the browser developer tools show any error relating to the plot contents? Are you able to right click the plot tabs and open in a new browser tab and see it working there?
This might be down to a quirk of how our network is set up, in which case, feel free to ignore this. Seems strange though!
This likely is the case, but it would be nice to get to the bottom of what is causing it.
Does the browser developer tools show any error relating to the plot contents? Are you able to right click the plot tabs and open in a new browser tab and see it working there?
Loading the proxy url directly does bring up the page, but it's generally blank (except for the info tab, which works). Inspecting the dev console, I see:
[bokeh] Lost websocket 0 connection, 1006 ()
[bokeh] Websocket connection 0 disconnected, will not attempt to reconnect
Interesting, we've seen similar issues in a few other places where whatever proxy is being placed in front of Jupyter is not correctly handling websocket connections.
our jupyter setup runs a bit differently
Are you able to share any more detail on how you are accessing Jupyter? Specifically what proxying technology is being used.
Am I correct in thinking the regular dashboard works via the /proxy/... url but the inidividual plots do not?
Are you able to share any more detail on how you are accessing Jupyter? Specifically what proxying technology is being used.
We use Open OnDemand for this, but I don't know much more than that, sorry!
Am I correct in thinking the regular dashboard works via the
/proxy/...url but the inidividual plots do not?
That's right -- the menu at the top shows up, but all tabs (except info) are blank / nonexistent.
That's right -- the menu at the top shows up, but all tabs (except info) are blank / nonexistent
Ok it sounds like all the websocket connections are being dropped, which is how we update the plots. I suspect that Open OnDemand is proxying the HTTP traffic and perhaps not handling the websocket connections correctly.
Jupyter also uses websockets, so things must be working correctly in that case. I'm not sure what is going wrong here though.
We've seen a similar problem in dask/dask#5432 on AWS Sagemaker.
Looks like this also came up in #176.
Out of curiosity @bmcfee what bokeh version is in your environment?
Out of curiosity @bmcfee what bokeh version is in your environment?
2.3.0.
Here's a full conda env
Singularity> conda list
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(139): Could not remove or rename /ext3/miniconda3/conda-meta/dask-core-2021.3.1-pyhd8ed1ab_0.json. Please remove this file manually (you may need to reboot to free file handles)
# packages in environment at /ext3/miniconda3:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
aiohttp 3.7.4.post0 pypi_0 pypi
anyio 2.2.0 pypi_0 pypi
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
argon2-cffi 20.1.0 pypi_0 pypi
async-generator 1.10 pypi_0 pypi
async-timeout 3.0.1 pypi_0 pypi
atk-1.0 2.36.0 h3371d22_4 conda-forge
attrs 20.3.0 pypi_0 pypi
babel 2.9.0 pypi_0 pypi
backcall 0.2.0 pypi_0 pypi
bleach 3.3.0 pypi_0 pypi
blosc 1.21.0 h9c3ff4c_0 conda-forge
bokeh 2.3.0 pypi_0 pypi
brotli 1.0.9 h9c3ff4c_4 conda-forge
brotlipy 0.7.0 py38h497a2fe_1001 conda-forge
brunsli 0.1 h9c3ff4c_0 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.17.1 h7f98852_1 conda-forge
ca-certificates 2020.12.5 ha878542_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cairo 1.16.0 h6cf1ce9_1008 conda-forge
certifi 2020.12.5 py38h578d9bd_1 conda-forge
cffi 1.14.5 py38ha65f79e_0 conda-forge
chardet 4.0.0 py38h578d9bd_1 conda-forge
charls 2.2.0 h9c3ff4c_0 conda-forge
click 7.1.2 pypi_0 pypi
cloudpickle 1.6.0 py_0 conda-forge
conda 4.9.2 py38h578d9bd_0 conda-forge
conda-package-handling 1.7.2 py38h8df0ef7_0 conda-forge
cryptography 3.4.6 py38ha5dfef3_0 conda-forge
cycler 0.10.0 py_2 conda-forge
cytoolz 0.11.0 py38h497a2fe_3 conda-forge
dask 2021.1.1 pypi_0 pypi
dask-jobqueue 0.7.2 pypi_0 pypi
dask-labextension 5.0.1 pypi_0 pypi
dbus 1.13.6 hfdff14a_1 conda-forge
decorator 4.4.2 py_0 conda-forge
defusedxml 0.7.1 pypi_0 pypi
distributed 2021.1.1 pypi_0 pypi
entrypoints 0.3 pypi_0 pypi
expat 2.2.10 h9c3ff4c_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 2.001 hab24e00_0 conda-forge
font-ttf-source-code-pro 2.030 hab24e00_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.13.1 hba837de_1004 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
fribidi 1.0.10 h36c2ea0_0 conda-forge
fsspec 0.8.7 pyhd8ed1ab_0 conda-forge
gdk-pixbuf 2.42.4 h04a7f16_2 conda-forge
gettext 0.19.8.1 h0b5b191_1005 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
glib 2.68.0 h9c3ff4c_2 conda-forge
glib-tools 2.68.0 h9c3ff4c_2 conda-forge
graphite2 1.3.13 h58526e2_1001 conda-forge
graphviz 2.47.0 h93c640b_0 conda-forge
gst-plugins-base 1.18.4 h29181c9_0 conda-forge
gstreamer 1.18.4 h76c114f_0 conda-forge
gtk2 2.24.33 hab0c2f8_0 conda-forge
gts 0.7.6 h64030ff_2 conda-forge
h5py 3.1.0 nompi_py38hafa665b_100 conda-forge
harfbuzz 2.8.0 h83ec7ef_1 conda-forge
hdf5 1.10.6 nompi_h6a2412b_1114 conda-forge
heapdict 1.0.1 pypi_0 pypi
icu 68.1 h58526e2_0 conda-forge
idna 2.10 pyh9f0ad1d_0 conda-forge
imagecodecs 2021.1.28 py38h5b4e65a_0 conda-forge
imageio 2.9.0 py_0 conda-forge
ipykernel 5.5.0 pypi_0 pypi
ipython 7.22.0 pypi_0 pypi
ipython-genutils 0.2.0 pypi_0 pypi
ipywidgets 7.6.3 pypi_0 pypi
jedi 0.18.0 pypi_0 pypi
jinja2 2.11.3 pypi_0 pypi
jpeg 9d h36c2ea0_0 conda-forge
json5 0.9.5 pypi_0 pypi
jsonschema 3.2.0 pypi_0 pypi
jupyter 1.0.0 pypi_0 pypi
jupyter-bokeh 3.0.0 pypi_0 pypi
jupyter-client 6.1.12 pypi_0 pypi
jupyter-console 6.4.0 pypi_0 pypi
jupyter-core 4.7.1 pypi_0 pypi
jupyter-packaging 0.7.12 pypi_0 pypi
jupyter-server 1.5.1 pypi_0 pypi
jupyter-server-proxy 3.0.2 pypi_0 pypi
jupyterlab 3.0.12 pypi_0 pypi
jupyterlab-pygments 0.1.2 pypi_0 pypi
jupyterlab-server 2.3.0 pypi_0 pypi
jupyterlab-widgets 1.0.0 pypi_0 pypi
jxrlib 1.1 h7f98852_2 conda-forge
kiwisolver 1.3.1 py38h1fd1430_1 conda-forge
krb5 1.17.2 h926e7f8_0 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.35.1 hea4e1c9_2 conda-forge
lerc 2.2.1 h9c3ff4c_0 conda-forge
libaec 1.0.4 h9c3ff4c_1 conda-forge
libblas 3.9.0 8_openblas conda-forge
libcblas 3.9.0 8_openblas conda-forge
libclang 11.1.0 default_ha53f305_0 conda-forge
libcurl 7.75.0 hc4aaa36_0 conda-forge
libdeflate 1.7 h7f98852_5 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libevent 2.1.10 hcdb4288_3 conda-forge
libffi 3.3 h58526e2_2 conda-forge
libgcc-ng 9.3.0 h2828fa1_18 conda-forge
libgd 2.3.0 h47910db_1 conda-forge
libgfortran-ng 9.3.0 hff62375_18 conda-forge
libgfortran5 9.3.0 hff62375_18 conda-forge
libglib 2.68.0 h3e27bee_2 conda-forge
libgomp 9.3.0 h2828fa1_18 conda-forge
libiconv 1.16 h516909a_0 conda-forge
liblapack 3.9.0 8_openblas conda-forge
libllvm11 11.1.0 hf817b99_0 conda-forge
libnghttp2 1.43.0 h812cca2_0 conda-forge
libopenblas 0.3.12 pthreads_h4812303_1 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libpq 13.1 hfd2b0eb_2 conda-forge
librsvg 2.50.3 hfa39831_1 conda-forge
libssh2 1.9.0 ha56f1ee_6 conda-forge
libstdcxx-ng 9.3.0 h6de172a_18 conda-forge
libtiff 4.2.0 hdc55705_0 conda-forge
libtool 2.4.6 h58526e2_1007 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libuv 1.41.0 h7f98852_0 conda-forge
libwebp 1.2.0 h3452ae3_0 conda-forge
libwebp-base 1.2.0 h7f98852_2 conda-forge
libxcb 1.13 h7f98852_1003 conda-forge
libxkbcommon 1.0.3 he3ba5ed_0 conda-forge
libxml2 2.9.10 h72842e0_3 conda-forge
libzopfli 1.0.3 h9c3ff4c_0 conda-forge
locket 0.2.1 pypi_0 pypi
lz4-c 1.9.3 h9c3ff4c_0 conda-forge
lzo 2.10 h516909a_1000 conda-forge
markupsafe 1.1.1 pypi_0 pypi
matplotlib 3.3.4 py38h578d9bd_0 conda-forge
matplotlib-base 3.3.4 py38h0efea84_0 conda-forge
mistune 0.8.4 pypi_0 pypi
mock 4.0.3 py38h578d9bd_1 conda-forge
msgpack 1.0.2 pypi_0 pypi
multidict 5.1.0 pypi_0 pypi
mysql-common 8.0.23 ha770c72_1 conda-forge
mysql-libs 8.0.23 h935591d_1 conda-forge
nbclassic 0.2.6 pypi_0 pypi
nbclient 0.5.3 pypi_0 pypi
nbconvert 6.0.7 pypi_0 pypi
nbformat 5.1.2 pypi_0 pypi
ncurses 6.2 h58526e2_4 conda-forge
nest-asyncio 1.5.1 pypi_0 pypi
networkx 2.5 py_0 conda-forge
nodejs 15.12.0 h92b4a50_0 conda-forge
notebook 6.3.0 pypi_0 pypi
nspr 4.30 h9c3ff4c_0 conda-forge
nss 3.63 hb5efdd6_0 conda-forge
numexpr 2.7.3 py38h51da96c_0 conda-forge
numpy 1.20.2 pypi_0 pypi
olefile 0.46 pyh9f0ad1d_1 conda-forge
openjpeg 2.4.0 hf7af979_0 conda-forge
openssl 1.1.1k h7f98852_0 conda-forge
packaging 20.9 pyh44b312d_0 conda-forge
pandas 1.2.3 pypi_0 pypi
pandocfilters 1.4.3 pypi_0 pypi
pango 1.42.4 h69149e4_5 conda-forge
parso 0.8.1 pypi_0 pypi
partd 1.1.0 py_0 conda-forge
pcre 8.44 he1b5a44_0 conda-forge
pexpect 4.8.0 pypi_0 pypi
pickleshare 0.7.5 pypi_0 pypi
pillow 8.1.2 py38ha0e1e83_0 conda-forge
pip 21.0.1 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
pooch 1.3.0 pyhd8ed1ab_0 conda-forge
prometheus-client 0.9.0 pypi_0 pypi
prompt-toolkit 3.0.18 pypi_0 pypi
psutil 5.8.0 pypi_0 pypi
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pypi_0 pypi
pycosat 0.6.3 py38h497a2fe_1006 conda-forge
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pygments 2.8.1 pypi_0 pypi
pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pyqt 5.12.3 py38h578d9bd_7 conda-forge
pyqt-impl 5.12.3 py38h7400c14_7 conda-forge
pyqt5-sip 4.19.18 py38h709712a_7 conda-forge
pyqtchart 5.12 py38h7400c14_7 conda-forge
pyqtwebengine 5.12.1 py38h7400c14_7 conda-forge
pyrsistent 0.17.3 pypi_0 pypi
pysocks 1.7.1 py38h578d9bd_3 conda-forge
pytables 3.6.1 py38hc386592_3 conda-forge
python 3.8.5 h7579374_1
python-dateutil 2.8.1 py_0 conda-forge
python-graphviz 0.16 pyh243d235_2 conda-forge
python_abi 3.8 1_cp38 conda-forge
pytz 2021.1 pyhd8ed1ab_0 conda-forge
pywavelets 1.1.1 py38h5c078b8_3 conda-forge
pyyaml 5.4.1 pypi_0 pypi
pyzmq 22.0.3 pypi_0 pypi
qt 5.12.9 hda022c4_4 conda-forge
qtconsole 5.0.3 pypi_0 pypi
qtpy 1.9.0 pypi_0 pypi
readline 8.0 he28a2e2_2 conda-forge
requests 2.25.1 pyhd3deb0d_0 conda-forge
ruamel_yaml 0.15.80 py38h497a2fe_1004 conda-forge
scikit-image 0.18.1 py38h51da96c_0 conda-forge
scipy 1.6.1 py38hb2138dd_0 conda-forge
send2trash 1.5.0 pypi_0 pypi
setuptools 54.2.0 pypi_0 pypi
simpervisor 0.4 pypi_0 pypi
six 1.15.0 pyh9f0ad1d_0 conda-forge
snakeviz 2.1.0 pyh9f0ad1d_0 conda-forge
snappy 1.1.8 he1b5a44_3 conda-forge
sniffio 1.2.0 pypi_0 pypi
sortedcontainers 2.3.0 pypi_0 pypi
sqlite 3.35.3 h74cdb3f_0 conda-forge
tblib 1.7.0 pypi_0 pypi
terminado 0.9.3 pypi_0 pypi
testpath 0.4.4 pypi_0 pypi
tifffile 2021.3.17 pyhd8ed1ab_0 conda-forge
tk 8.6.10 h21135ba_1 conda-forge
toolz 0.11.1 py_0 conda-forge
tornado 6.1 py38h497a2fe_1 conda-forge
tqdm 4.59.0 pyhd8ed1ab_0 conda-forge
traitlets 5.0.5 pypi_0 pypi
typing-extensions 3.7.4.3 pypi_0 pypi
urllib3 1.26.4 pyhd8ed1ab_0 conda-forge
wcwidth 0.2.5 pypi_0 pypi
webencodings 0.5.1 pypi_0 pypi
wheel 0.36.2 pyhd3deb0d_0 conda-forge
widgetsnbextension 3.5.1 pypi_0 pypi
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.7.0 h7f98852_0 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h7f98852_1 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h516909a_0 conda-forge
yarl 1.6.3 pypi_0 pypi
zfp 0.5.5 h9c3ff4c_4 conda-forge
zict 2.0.0 pypi_0 pypi
zlib 1.2.11 h516909a_1010 conda-forge
zstd 1.4.9 ha95c52a_0 conda-forge
Thanks @bmcfee that rule it out as a 1.x issue.
My worry here is that it is an issue being caused by the Open OnDemand proxy which we have no control over.