pgo is not compatible with k8s v1.25

[youvegotmoxie]

Overview

After upgrading to Kubernetes v1.25.0 the pgo pod is crashing because the CronJob batch/v1beta1 API is no longer served. The code for the operator needs to be updated to (also) look for the batch/v1 apiVersion which has been GA since k8s v1.21

$ k get pod -n postgres-operator
NAME                           READY   STATUS             RESTARTS      AGE
pgo-6d89f7c7cb-4dhkf           0/1     CrashLoopBackOff   7 (68s ago)   12m
pgo-upgrade-6f956b867c-4tzts   1/1     Running            0             12m

I am using the latest version of the postgres-operator Helm chart from here

Environment

Please provide the following details:

• Platform: Kubernetes
• Platform Version: v1.25.0

Steps to Reproduce

1. Install the latest version of the PGO
2. Upgrade Kubernetes to v1.25.0
3. Observe that the pgo pod is now crash looping

EXPECTED

1. To not crash after the k8s upgrade

ACTUAL

1. pgo pod goes into CrashLoopBackoff due to the beta CronJob API not being served

Logs

time="2022-08-29T15:33:28Z" level=error msg="if kind is a CRD, it should be installed before calling Start" error="no matches for kind \"CronJob\" in version \"batch/v1beta1\"" file="sigs.k8s.io/[email protected]/pkg/log/deleg.go:144" func="log.(*DelegatingLogger).Error" kind=CronJob.batch version=5.1.3-0
panic: no matches for kind "CronJob" in version "batch/v1beta1"

goroutine 1 [running]:
main.assertNoError(...)
        github.com/crunchydata/postgres-operator/cmd/postgres-operator/main.go:40
main.main()
        github.com/crunchydata/postgres-operator/cmd/postgres-operator/main.go:100 +0x550

Furthermore, while CronJobs are required for backups, they aren't critical for managing the lifecycle of the database cluster from what I can tell. It may be better to emit a warning and skip trying to reconcile the CronJob(s) objects instead of crashing the application.

[ButeForce]

I have the same issue , also when installing fresh installtion on kubernetes 1.25 it fails to create CDR , it says :

The CustomResourceDefinition "postgresclusters.postgres-operator.crunchydata.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

[jelmer]

I have the same issue , also when installing fresh installtion on kubernetes 1.25 it fails to create CDR , it says :

The CustomResourceDefinition "postgresclusters.postgres-operator.crunchydata.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

Try passing --server-side to kubectl; that should address it.

[jelmer]

I've added support for Kubernetes 1.25 in #3370; experimental container build from those changes available at ghcr.io/jelmer/postgres-operator:rocky9-v4.5.0-855-g1b8413409

[andrewlecuyer]

I just wanted to note on this thread that we are actively working on 1.25 compatibility/support, which will be included the next release of PGO.

[tjmoore4]

@youvegotmoxie as mentioned above, PR https://github.com/CrunchyData/postgres-operator/pull/3370 should resolve these issues and is planned to be available in the next release.

[loxy]

When is the next release, approximately?

[MrFox131]

When is the next release scheduled?

[yanmxa]

Wait for the next release.

[sorinpepelea]

any updates for the release of this fix ? @jelmer published a nice fix few month ago, but this repo is yet to focus on older version of kubernetes. this is slowly becoming unusable.

[jansmets]

dead in the water ... have been waiting almost 3 months now for a new release.

[zevisert]

Hey PGO team (@tjmoore4, @andrewlecuyer, @cbandy, @benjaminjb, I think that's some of you), people are obviously getting frustrated here, so is there anything the community can do to help move a release with #3370 forward?

If there's other issues that you need to solve before a release is possible, please let us know! I'm sure some of us would be willing to help out!

[andrewlecuyer]

We are currently planning a mid-December release for PGO v5.3.0, which will include the changes required for Kubernetes v1.25 compatibility.

I also wanted to point out the Crunchy Postgres for Kubernetes release policy, which specifically highlights our standard release cadence across each quarter. For instance, as described in the policy:

Crunchy Data anticipates the following release schedule for Crunchy PostgreSQL for Kubernetes:

• At least one (1) PATCH release per quarter, timed around the PGDG Release
• At least one (1) MINOR release per quarter, timed around the end-of-the-quarter

PGO v5.3.0 is therefore the new MINOR version of PGO that we're looking to release this quarter.

Thanks for your feedback, and we appreciate your patience as we look to get this update released in the coming weeks!

[controlpl4n3]

Since I'm in no hurry to use PGO, I'll just delete the deployment. I would rather not have the pod in crash loop backoff for the next month.

[benjaminjb]

PGO v5.3.0 has been released! It is compatible with K8s v1.25, as noted above.

[Zvezdoreel]

Will the community operator in the openshift operatorhub supposed to get this update as well? I can see that only the certified version is currently v5.3.0. (Nor the community neither the marketplace)

[tjmoore4]

@Zvezdoreel We are planning to release that version as well, but are currently encountering some issues deploying our installers via the Red Hat certification process. We are actively working with Red Hat on a resolution and hope to have all OLM installers available soon.

[haf]

So now this is biting me, because 5.3.0 doesn't have working backups: #3549

So I guess the conclusion is that Crunchy Data has no support for Kubernetes 1.25?

[andrewlecuyer]

To clarify, PGO is fully compatible with Kubernetes 1.25.

The issue @haf references is unrelated, and cannot be reproduced (and is being addressed/discussed in https://github.com/CrunchyData/postgres-operator/issues/3549). Additionally, backups have been confirmed to work properly with the v5.3.0 release.