pangeo-cloud-federation icon indicating copy to clipboard operation
pangeo-cloud-federation copied to clipboard

Published 20 hours ago • verified publisher icon pangeo-data

Help me upgrade image for ocean.pangeo.io

Open rabernat opened this issue 4 years ago • 3 comments

Ocean.pangeo.io, by far our most heavily used cluster, is running Dockerfile pangeo/pangeo-notebook-onbuild:2019.12.08. Our stack is evolving and it has fallen behind quite a bit.

Because we have eliminated the notebook onbuild docker image (xref https://github.com/pangeo-data/pangeo-docker-images/issues/60), bringing this hub up to speed will be a bit tricky. I imagine we will need to:

  • Create a new base environment based on the pangeo-notebook image
  • Merge that environment.yml with the current ocean environment.yml

I am a bit nervous that we will get lots of package conflicts, but maybe not.

Since I have been away from active hub maintenance for months and fallen behind on many technical details, I would sincerely appreciate some help in performing this upgrade. Perhaps we can schedule an hour this coming week where some of the experts (e.g. @tjcrone, @TomAugspurger, @scottyhq) are online and can provide some support as I try a few things.

Since this would be a major change in environment, we might also want to make some kind of announcement to users (see #595).

rabernat avatar May 04 '20 01:05 rabernat

Happy to help out. I'm relatively free today and Wednesday but I can make other days work as well.

I think the best option for ocean is to

  1. Update the Dockerfile to use FROM pangeo/base-image
  2. Update the environment.yaml file to include the pangeo-notebook metapackage to mange the dask and Jupyter components.
  3. Fill out the environment.yaml with ocean-specific packages (essentially the contents of the current environment.yaml)

I don't know that we should use the pangeo-notebook image. I think that if we're customizing the conda environment with ocean-specific packages, we should just own the entire environment.

TomAugspurger avatar May 04 '20 14:05 TomAugspurger

I don't know that we should use the pangeo-notebook image. I think that if we're customizing the conda environment with ocean-specific packages, we should just own the entire environment.

I think we're on the same page. When I said "based on the pangeo-notebook image", I meant, using that as a template.

I'll get a PR started.

rabernat avatar May 04 '20 17:05 rabernat

You can use https://github.com/pangeo-data/pangeo-cloud-federation/tree/staging/deployments/icesat2/image/binder as a model. No need for a conda-linux-64.lock if you don't want to generate one. You can stick with just environment.yml

scottyhq avatar May 06 '20 17:05 scottyhq