Preparing to upgrade to OpenShift Container Platform 4.9 [Kubernetes 1.22] APIs Deprecated

[stephenkehoe]

The next OpenShift Container Platform release, currently planned to be OpenShift Container Platform 4.9, is expected to use Kubernetes 1.22. Kubernetes 1.22 removed a significant number of deprecated v1beta1 APIs.

OpenShift Container Platform 4.8.14 introduced a requirement that an administrator must provide a manual acknowledgment before the cluster can be upgraded from OpenShift Container Platform 4.8 to 4.9. This is to help prevent issues after upgrading to OpenShift Container Platform 4.9, where APIs that have been removed are still in use by workloads, tools, or other components running on or interacting with the cluster. Administrators must evaluate their cluster for any APIs in use that will be removed and migrate the affected components to use the appropriate new API version. After this is done, the administrator can provide the administrator acknowledgment.

All clusters require this administrator acknowledgment before they can be upgraded to OpenShift Container Platform 4.9.

Removed Kubernetes APIs

Kubernetes 1.22 removed the following deprecated v1beta1 APIs. If your cluster, or any idle workloads or tools use any of these APIs, you must migrate them to the appropriate new version before upgrading to OpenShift Container Platform 4.9.

Resource	API	Notable changes
APIService	apiregistration.k8s.io/v1beta1	No
CertificateSigningRequest	certificates.k8s.io/v1beta1	Yes
ClusterRole	rbac.authorization.k8s.io/v1beta1	No
ClusterRoleBinding	rbac.authorization.k8s.io/v1beta1	No
CSIDriver	storage.k8s.io/v1beta1	No
CSINode	storage.k8s.io/v1beta1	No
CustomResourceDefinition	apiextensions.k8s.io/v1beta1	Yes
Ingress	extensions/v1beta1	Yes
Ingress	networking.k8s.io/v1beta1	Yes
IngressClass	networking.k8s.io/v1beta1	No
Lease	coordination.k8s.io/v1beta1	No
LocalSubjectAccessReview	authorization.k8s.io/v1beta1	Yes
MutatingWebhookConfiguration	admissionregistration.k8s.io/v1beta1	Yes
PriorityClass	scheduling.k8s.io/v1beta1	No
Role	rbac.authorization.k8s.io/v1beta1	No
RoleBinding	rbac.authorization.k8s.io/v1beta1	No
SelfSubjectAccessReview	authorization.k8s.io/v1beta1	Yes
StorageClass	storage.k8s.io/v1beta1	No
SubjectAccessReview	authorization.k8s.io/v1beta1	Yes
TokenReview	authentication.k8s.io/v1beta1	No
ValidatingWebhookConfiguration	admissionregistration.k8s.io/v1beta1	Yes
VolumeAttachment	storage.k8s.io/v1beta1	No

[stephenkehoe]

https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/

[michaelsauter]

@stephenkehoe Thanks a lot for this! One question: where did you get the above list? In the linked blog post, I do not see any mention of changes to e.g. RoleBinding. Is there another resource to read up on those changes?

The changes listed in the blog post sound safe to me, nothing comes to mind that affects what ODS is doing.

[gerardcl]

hi! I have some updates regarding this topic:

1. See "Preparing to upgrade to OpenShift Container Platform 4.9" knowledge page from redhat here

2. From (1), I have checked on our OSD4.8 cluster what k8s API resources might be affected, by running command oc get apirequestcounts, and filtering by existing REMOVEDINRELEASE we see:

$ oc get apirequestcount
NAME                                                                           REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
certificatesigningrequests.v1beta1.certificates.k8s.io                         1.22               0                       0
clusterrolebindings.v1beta1.rbac.authorization.k8s.io                          1.22               0                       0
customresourcedefinitions.v1beta1.apiextensions.k8s.io                         1.22               12                      292
flowschemas.v1alpha1.flowcontrol.apiserver.k8s.io                              1.21               0                       0
ingresses.v1beta1.extensions                                                   1.22               18                      437
ingresses.v1beta1.networking.k8s.io                                            1.22               0                       0
mutatingwebhookconfigurations.v1beta1.admissionregistration.k8s.io             1.22               0                       0
rolebindings.v1beta1.rbac.authorization.k8s.io                                 1.22               0                       0
roles.v1beta1.rbac.authorization.k8s.io                                        1.22               0                       0
validatingwebhookconfigurations.v1beta1.admissionregistration.k8s.io           1.22               0                       0

3. Based on results in (2) we have checked what is doing requests to those used resources (we have checked two days in a row already):

• customresourcedefinitions.v1beta1.apiextensions.k8s.io, used by: -- userAgent: openshift-pipelines-operator/v0.0.0 -- userAgent: velero/v1.7.1-konveyor

• ingresses.v1beta1.extensions, used by: -- userAgent: cluster-policy-controller/v0.0.0 -- userAgent: kube-controller-manager/v1.21.6+4b61f94 -- userAgent: velero-server/v1.7.1-konveyor

4. From this solution article by RedHat, we can be pretty sure that we are not affected by changes in ingresses.v1beta1.extensions, and we believe regarding customresourcedefinitions.v1beta1.apiextensions.k8s.io we are also in a good shape since we might get a newest/working openshift-pipelines-operator version.

5. Nevertheless, we have an open ticket to double check such assumptions with RedHat, and once we have all fully verified we will be able to upgrade.

Finally, please also take into account we do use both ods-core and ods-pipeline ODS frameworks in that cluster (and some other operators). Hence we can use all the information provided here as a baseline for all clusters using ODS that require upgrading.

[michaelsauter]

Regarding OpenShift Pipelines:

See https://docs.openshift.com/container-platform/4.9/cicd/pipelines/op-release-notes.html#compatibility-support-matrix_op-release-notes. Since we are on 4.8, we use OpenShift Pipelines 1.5, which is based on Tekton 0.24.x. According to Tekton 0.27.3 is the first release with support for Kubernetes 1.22 (https://github.com/tektoncd/pipeline/releases/tag/v0.27.3). So the update to OpenShift Pipelines 1.6 should resolve the issue as it is based on 0.28.x.

[cschweikert]

Regarding Kubernetes API there might be relevant dependencies towards Kubernetes-related Jenkins plugins:

• kubernetes:1.31.0
• kubernetes-client-api:5.10.1-171.vaa0774fb8c20
• kubernetes-credentials:0.9.0

I suspect those plugins to be responsible for https://github.com/opendevstack/ods-core/issues/1123. Maybe this is also relevant here.

[cschweikert]

Jenkins plugins kubernetes of version 1.30.6+ requires Kubernetes version 1.19+. See also https://github.com/jenkinsci/kubernetes-plugin/releases/tag/kubernetes-1.30.6

What version of Kubernetes are we running on right now?

[michaelsauter]

@cschweikert the cluster you used is OpenShift 4.8, which uses Kubernetes 1.21. You can always check by clicking on the "question mark" icon in the header, and then selecting "About" from the menu.