Discussion: Image Management & Staging -> Prod workflow

[scottyhq]

This is an issue for discussing the current staging -> prod workflow and management and testing of Docker images.

The original design of hubploy works great for a single image per hub, defined in a subfolder. This is currently the configuration for the GCP hub: https://github.com/pangeo-data/pangeo-cloud-federation/tree/staging/deployments/gcp-uscentral1b/image/binder

Over time on the aws-uswest2.pangeo.io we've found it helpful to have a handful of different images available for selection. In particular, pangeo/ml-notebook which has specific GPU config https://github.com/pangeo-data/pangeo-cloud-federation/blob/7e842bf024d0aa90d42c8f0e3a6fa79bc0e4b91f/deployments/icesat2/config/common.yaml#L31-L39

Or images for special events like hackweeks: https://github.com/pangeo-data/pangeo-cloud-federation/blob/7e842bf024d0aa90d42c8f0e3a6fa79bc0e4b91f/deployments/icesat2/config/common.yaml#L19-L22

A while back I started using the latest tag in the profile list because I couldn't be bothered updating tags frequently. This works pretty well if the 'latest' tag is always guaranteed as a tested production release. Which is the goal with our pangeo-docker-images repository design (https://github.com/pangeo-data/pangeo-docker-images#image-tagging-and-continuous-building). This key to using latest tags in config is to ensure that the pod always re-pulls the image (default is 'IfNotPresent') when launched:

    singleuser:
      image:
         pullPolicy: 'Always'

This still isn't ideal from a reproducibility standpoint (probably best to use explicit tags).

Possible next steps:

1. Tackle this open issue on hubploy for multiple images support https://github.com/yuvipanda/hubploy/issues/10 ?

2. Add GitHub Action to watch for new tags on DockerHub and issue PRs to bump all image tags under profile list?

[salvis2]

This would be really nice. We could probably quash the issue we had a bit ago with https://github.com/yuvipanda/hubploy/issues/73 and have some listed images from DockerHub, some from ECR, etc.