Search for available free ports if default port 8000 is not available

[davidrpugh]

Thanks for the awesome tool! We are starting to use jupyterlab-nvdashboard on a multi-user HPC system. I am trying to figure out what will happen if two user jobs land on the same compute node and both users run the server in their job scripts as follows.

python -m jupyterlab_nvdashboard.server &

Will the first user get the default port 8000 and then the second user get an error or warning? Or will the jupyterlab_nvdashboard.server look for available ports?

[jacobtomlinson]

Thanks for raising this! You're right that the first user will get the port 8000 and the second will get an error.

As a workaround I expect the second user could specify the port. But I appreciate this isn't ideal.

python -m jupyterlab_nvdashboard.server 8001 &

It would be nice to detect this and switch to a random high port if 8000 is taken.

[davidrpugh]

Is nvdashboard subclassing most of its core functionality from Bokeh? Should this type of logic be submitted as a PR upstream? Am happy to contribute a PR with a little guidance as to where to start.

[jacobtomlinson]

We use Bokeh to run the server, however I'm not sure that falling back to a random high port is going to be generic enough for this to make sense as an upstream change.

I expect an exception will be raised somewhere around here when trying to run multiple dashboards on the default port. We should catch this exception and try again with a random high port while logging a warning.

[owenlamont]

Can this problem can be reproduced by running nvdashboard from two Python environments on the same system at the same time?

[davidrpugh]

I wonder if we can use the same trick that Tensorboard user and just try to set port=0.

https://github.com/tensorflow/tensorboard/issues/1177

I have tried launching a server with port=0 and it launches and after some quick inspection using netstat -ltnp I was able to find that the server was launched at some random open port.

It would appear that this functionality already exists! Now what needs to happen is we need to check if the user passes 0 as the port number and then capture the allocated port and return it in a nice log message.

[davidrpugh]

The following will launch the Bash magic that extracts the port number using netstat. Seems brittle though and am not sure how to go about doing this from within Python.

python -m jupyterlab_nvdashboard.server 0 &
NVDASHBOARD_PID=$!
NVDASHBOARD_PORT=$( netstat -pl | grep $NVDASHBOARD_PID | tail -n 1 | awk -F "[ :]+" '{print $6}' )
echo "Your NVDashboard Server is running at $( hostname -f ):$NVDASHBOARD_PORT"