[BUG] - Dependency conflicts with Jupyterlab pod and visualization conda environments

[Adam-D-Lewis]

Describe the bug

The jupyterlab pod includes several dependencies which I believe are conflicting at times with the dependencies in conda environments especially those used for visualization.

For Nebari version 2023.10.1, I needed to pin these dependency versions to get the visualization environment to work as expected.

channels:
  - conda-forge
dependencies:
  - python==3.11.7
  - ipympl==0.9.3
  - matplotlib==3.8.2
  - jupyterlab==3.6.6
  - jupyterlab_widgets==1.1.7
  - jupyterlab_server==2.25.0
  - jupyter_client==8.6.0
  - ipykernel==6.26.0
  - pandas
  - numpy
  - pyarrow
  - pyarrow
  - pytz
  - plotly
  - bokeh
  - nbformat
  - ipympl
  - holoviews
  - hvplot
  - azure-storage-blob
  - ipython
  - polars
  - datashader
  - numdifftools

For Nebari version 2024.1.1, I needed to pin these environments to get this environment to work as expected.

channels:
  - conda-forge
dependencies:
  - python==3.10.13
  - ipympl==0.9.3
  - matplotlib==3.8.2
  - jupyterlab==4.0.12
  - jupyterlab_widgets==3.0.9
  - jupyterlab_server==2.25.2
  - jupyter_client==8.6.0
  - ipykernel==6.29.0
  - pandas
  - numpy
  - pyarrow
  - pyarrow
  - pytz
  - plotly
  - bokeh
  - nbformat
  - ipympl
  - holoviews
  - hvplot
  - azure-storage-blob
  - ipython
  - polars
  - datashader
  - numdifftools

When I didn't pin the dependencies, the plotting notebooks produced blank cell outputs. E.g. something like

import hvplot.pandas
import holoviews as hv

df = ...

myplot = df.hvplot(x='x', y=['y', 'z'], color=['blue', 'red'], rasterize=True)

# Show the plot
hv.extension('bokeh')
myplot

would plot nothing. Pinning the dependencies fixed the issue.

The way I determined which versions I should pin were to pull the Nebari jupyterlab docker image specified in the nebari-config.yaml file, activate the default conda environment and run conda list | grep -e python -e ipympl -e matplotlib -e jupyterlab -e jupyterlab_widgets -e jupyterlab_server -e jupyter_client -e ipykernel. Those may not all need to be pinned, but it does work when I do pin all those.

It's tedious to have to update all the visualization environment dependencies when updating Nebari, and would be nice if this could be done in an automated way or if the dependency conflicts could be avoided which I think is more feasible now that JHub Apps has been added to Nebari.