Kubeflow 1.6 Dependency Versions

[annajung]

Based on the initial input gathered from different WGs and release team discussions, Kubeflow 1.6 will aim to support the following dependency versions.

• Kubernetes 1.22
• Istio 1.14.1
• Knative 1.2
• Cert Manager v1.5.0
• Dex v2.31.2

While different Kubernetes versions may be tested, our primary goal for the release is to focus our efforts on Kubernetes version 1.22.

If you have any concerns or like to propose a different version, please leave a comment here to allow transparency across all WG leads.

@kubeflow/wg-automl-leads @kubeflow/wg-manifests-leads @kubeflow/wg-training-leads @kubeflow/wg-pipeline-leads @kubeflow/wg-notebooks-leads @pvaneck @yuzisun @zijianjoy @kubeflow/release-team

For additional context, below are the versions supported in the previous Kubeflow 1.5 release

• Kubernetes 1.21, 1.20, 1.19
• Istio 1.11
• Knative 0.22.1
• Cert Manager v1.5.0
• Dex v2.22.0

[jbottum]

@annajung we might want to track the Kustomize version too. i.e. will it be 3.2 or 4.x

[annajung]

Based on the discussion from the May 10th community meeting, @yhwang raised concerns with the Knative version proposed here and shared Kserve's need to have at least Knative 1.0+

With Knative being used only by the Kserve WG, the release team plans to work with KServe team to figure out which version needs to be supported for the 1.6 release and update the proposal based on their feedback

cc @mstopa @pvaneck @yuzisun

[johnugeorge]

Ref: https://github.com/kubeflow/katib/pull/1828

Are we planning to use networking.istio.io/v1alpha3 or networking.istio.io/v1beta1 for VirtualService in 1.6 release ?

/cc @tenzen-y

[pvaneck]

We have decided to go with Knative 1.4 as the desired version of Knative for KF 1.6. I can help with with the initial PR updating the manifests.

[kimwnasptd]

Are we planning to use networking.istio.io/v1alpha3 or networking.istio.io/v1beta1 for VirtualService in 1.6 release?

@johnugeorge, @dpoulopoulos and I were testing the Istio 1.14 manifests with the whole Kubeflow without issues and using v1alpha3. To my best knowledge there shouldn't be an issue with v1alpha3 so I'd suggest to keep the current version for the first RC.

If later on we bump into issues with v1alpha3 we can include fixes in the release branches.

[annajung]

Hi @kimwnasptd could you also update us on the Istio version. During the release team meeting on June 27th, you mentioned that 1.13 is near EOL support which was a concern for using 1.13 for KF 1.6 release. Based on your comment above, has Manifest WG decided to use Istio 1.14 instead of 1.13 for KF 1.6?

[kimwnasptd]

@annajung @kubeflow/release-team, @dpoulopoulos and I have been testing Istio 1.14 until this point without seeing any issues.

We decided to go with Istio 1.13 instead of 1.14 since 1.13 will be EOL by October https://istio.io/latest/docs/releases/supported-releases/#support-status-of-istio-releases. The upgrade notes from 1.13 to 1.14 are also minimal so we are confident we won't bump into any issues https://istio.io/latest/news/releases/1.14.x/announcing-1.14/upgrade-notes/.

But we'll be keeping a close eye for any issues that might arise throughout the WGs during the feature freeze.

[kimwnasptd]

We have decided to go with Knative 1.4 as the desired version of Knative for KF 1.6. I can help with with the initial PR updating the manifests.

@pvaneck if you'd have some cycles to follow the instructions and update Knative that would be a huge help! If not I'll try to push for a PR early tomorrow for this one as well.

[juliusvonkohout]

We have decided to go with Knative 1.4 as the desired version of Knative for KF 1.6. I can help with with the initial PR updating the manifests.

@pvaneck if you'd have some cycles to follow the instructions and update Knative that would be a huge help! If not I'll try to push for a PR early tomorrow for this one as well.

it would be amazing if you could use tags instead of digests for the new knative images. Currently it is a mixture of both.

[annajung]

Tracking PRs:

• Kubernetes 1.22 https://github.com/kubeflow/manifests/pull/2230
• Istio 1.14.1 https://github.com/kubeflow/manifests/pull/2233
• Knative 1.4 https://github.com/kubeflow/manifests/pull/2231
• Cert Manager v1.5.0: https://github.com/kubeflow/manifests/pull/2076
• Dex v2.31.2: https://github.com/kubeflow/manifests/pull/2243

[annajung]

Update Knative 1.4 to 1.2: https://github.com/kubeflow/manifests/pull/2251

[mnmami]

@annajung will Kubeflow 1.6 keep support for Kubernetes versions 1.20 and 1.21?

[annajung]

Hi @mnmami sorry for the late reply! The release team has been working closely with manifest WG, who have been focused on testing against only the 1.22 k8s cluster. Therefore, we don't officially support 1.21 or 1.20 with the 1.6 release.

[mnmami]

@annajung thanks for clarifying. This information is very important for organizations to plan and manage their upgrade strategy for Kubeflow both on-prem and in the cloud.

[umka1332]

It would be very nice to update Seldon that is delivered with Kubeflow to support K8s 1.22+.

[sylus]

@kimwnasptd where can i find the instructions on how to upgrade knative properly from the previous CRDs

Just pointing to the new manifests isn't enough since the upgrade steps need to be performed and I thought kubeflow would offer these instructions in a centralized place even if just pointing to the knative upgrade steps which I think is just follow the release notes.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- github.com/kubeflow/manifests/common/knative/knative-serving/base?ref=v1.6.0
- github.com/kubeflow/manifests/common/knative/knative-eventing/base?ref=v1.6.0

[sylus]

Also the release blog for KF v1.6 claims that every component has been tested for 1.22?

https://blog.kubeflow.org/kubeflow-1.6-release/

However for spark the CRD calling v1beta1

https://github.com/kubeflow/manifests/blob/master/contrib/spark/spark-operator/base/sparkapplications.sparkoperator.k8s.io-crd.yaml#L1

I think Spark and Seldon aren't considering core components since I don't see them in testing suite:

https://github.com/kubeflow/manifests/pull/2230/files

[sylus]

This is what I needed to do to update knative:

 krm mutatingwebhookconfigurations.admissionregistration.k8s.io webhook.serving.knative.dev
 krm validatingwebhookconfigurations.admissionregistration.k8s.io validation.webhook.serving.knative.dev
 kubectl create -f ../../../serving-storage-version-migration.yaml (v0.18.3)
 kubectl create -f ../../../eventing-post-install.yaml (v0.23.0)
 // wait for job to finish
 kubectl delete -f ../../../eventing-post-install.yaml
 kubectl create -f ../../../eventing-post-install.yaml (v1.2.4)
 // wait for job to finish
 // re-sync ArgoCD

With this I was able to deploy Kubeflow on Azure AKS from v1.4.1 -> 1.6.0 via ArgoCD and everything seems to be working / sync'd except for seldon and spark.

[annajung]

Hi @umka1332 and @sylus, could you create a separate issue for Seldon and spark to be tracked outside of this issue? This issue was created to determine the core dependency versions for the 1.6 release. With the 1.6 release out, I'm closing this as it's been completed.

[umka1332]

Hi @umka1332 and @sylus, could you create a separate issue for Seldon and spark to be tracked outside of this issue? This issue was created to determine the core dependency versions for the 1.6 release. With the 1.6 release out, I'm closing this as it's been completed.

Done for Seldon - https://github.com/kubeflow/manifests/issues/2290