Image path of existing clusters when upgrading from pgo v4.7.5 to v4.7.6

[jufi1]

Overview

Initial situation

Until recently, we had a Postgres operator version 4.7.5 and a Postgres cluster version 13.7 running in our Kubernetes cluster.

What did we do

We have attempted to upgrade the operator from version 4.7.5 to version 4.7.6. Up to version 4.7.5 we pulled the pgo image from Dockerhub, which is no longer available there from version 4.7.6 onwards. We downloaded the resources for 4.7.6 and were also able to upgrade the PGO to the new version.

EXPECTED

When upgrading our current cluster with "pgo upgrade cluster_name" we would have expected the current cluster to apply the configurations from the new pgo-Version

PROBLEM

Instead of applying new config, our running cluster tries to rebuild the pods with the old Dockerhub image path. The new path is used when creating new clusters, but the old path is used for old clusters. We have already searched the configs of the PGO and the cluster itself and cannot find where the old path is stored, so we can change it. Maybe you can help us what we can do.

We haven't found any information on this issue tracker eg. so we are not sure if it's a problem with our environment or a general problem. Since we are in a corporate enviroment we are also not able to pull images directly, we have to use an artifact store.

Environment

Please provide the following details:

• Platform: (Kubernetes)
• Platform Version: (1.22)
• PGO Image Tag: (ubi8-13.7-4.7.6)
• Postgres Version (e.g. 13.7)
• Storage: (caas-default-sc csi.vsphere.vmware.com)

[rnwgnr]

Update:

• We tried to delete the cluster with pgo delete cluster cluster_name --keep-data --keep-backups to deploy a new cluster with the same name afterwards. Unfortunately the deletion fails. The delete job tries to create a pod/cluster_name-rmdata-... which fails as well due to referencing the old image path on Dockerhub. So at the moment we are not able to delete clusters that have been deployed before the image path change.
• Finally we managed to get a new cluster up and running by restoring from the latest backup of the existing cluster.

So the remaining question is:
What is the officially supported way to change the image path in an existing pgo deployment without extended downtime?

[Maghin]

So the remaining question is: What is the officially supported way to change the image path in an existing pgo deployment without extended downtime?

It might be to edit the spec.pgoimageprefix of the Pgcluster configuration :

kubectl edit Pgcluster/cluster_name
...
  pgoimageprefix: crunchydata
...

Then all pgo operation will use this image prefix so the deletion and the download of rmdata works.

Edit: Working with pgo:4.7.0 right now. oneline bash cmd : KUBE_EDITOR="sed -i 's/\(pgoimageprefix:.*$\)/pgoimageprefix: crunchydata/g'" kubectl edit Pgcluster cluster_name -n cluster_namespace

[benjaminjb]

Thank you for bringing this issue to our attention.

The PGO v4 line you are using is no longer available through the Crunchy Developer Program (for additional details about available versions of Crunchy Postgres for Kubernetes, please see the Supported Platforms page.

If you still require assistance with v4, please contact [email protected] to see if we can help further.