[ci] Also push to new_docker_prefix if present as part of AR migration

[daniel-goldstein]

Stacked on #12210, this alters build.py to push images to docker_prefix and to new_docker_prefix if present in the global config. This will make sure that when we switch docker_prefix to the new Artifact Registry that any in-use images are ready to go.

Once we are ready to switch over, we can

1. change docker_prefix to the new AR in the global-config
2. remove new_docker_prefix from the global-config
3. Revert this change

[daniel-goldstein]

Tested by running hailctl dev deploy -b hail-is/hail:main -s git_make_bash_image against a CI in my namespace and seeing that it succeeded and pushed an image here

[jigold]

Before we merge this, I'd like to have the new Artifact Registry in hail-vdc setup and have configured a cloud run job for the cleanup script. I think a daily run is good enough. https://github.com/GoogleCloudPlatform/gcr-cleaner/blob/main/docs/deploy-cloud-run.md

[jigold]

I used filters for the following images when I've run the Azure cleanup script, but we should double check these make sense still in light of changing how we use "cache" and there aren't any additional images or ones that we don't want to delete that are in this list:

  --filter 'auth:.*' \
  --filter 'base:.*' \
  --filter 'base_spark_3_2:.*' \
  --filter 'batch:.*' \
  --filter 'batch-driver-nginx:.*' \
  --filter 'batch-worker:.*' \
  --filter 'benchmark:.*' \
  --filter 'blog_nginx:.*' \
  --filter 'ci:.*' \
  --filter 'ci-intermediate:.*' \
  --filter 'ci-utils:.*' \
  --filter 'create_certs_image:.*' \
  --filter 'echo:.*' \
  --filter 'grafana:.*' \
  --filter 'hail-base:.*' \
  --filter 'hail-build:.*' \
  --filter 'hail-buildkit:.*' \
  --filter 'hail-run:.*' \
  --filter 'hail-run-tests:.*' \
  --filter 'hail-pip-installed-python37:.*' \
  --filter 'hail-pip-installed-python38:.*' \
  --filter 'hail-ubuntu:.*' \
  --filter 'memory:.*' \
  --filter 'monitoring:.*' \
  --filter 'notebook:.*' \
  --filter 'notebook_nginx:.*' \
  --filter 'prometheus:.*' \
  --filter 'service-base:.*' \
  --filter 'service-java-run-base:.*' \
  --filter 'test-ci:.*' \
  --filter 'test-monitoring:.*' \
  --filter 'test-benchmark:.*' \
  --filter 'website:.*' \

[daniel-goldstein]

Sounds good, I'll get this set up

[jigold]

One last thing -- do we need to change the Makefiles at all?

[daniel-goldstein]

The Makefiles grab the docker_prefix from the global-config each time they are run so once we make the change in the global-config nothing else needs to change.

Actually, I think this entire PR is not even needed. When CI does a deploy, the changes it applies to Kubernetes include fully qualified image names, e.g. gcr.io/hail-vdc/batch:asdf1234. If we were to swap out the docker_prefix global-config variable, CI would start to create new images that are pushed to the new repository (it would kill the cache for a single build but whatever), but the existing images would still exist and be undisrupted. The only images that need to exist in the new container registry when the switch is made are the images that we push on bootstrap which I am going to do manually anyway.

[daniel-goldstein]

Closing as this is not necessary.