KF Release 1.6 tracking

[annajung]

This issue will be used to provide high-level updates for the Kubeflow 1.6 release.

• Release Timeline
• Release Team
• Release Board

cc @kubeflow/release-team

[annajung]

• Docs tracking issue
• WG tracking issues
  • AutoML
  • Training Operator
  • KServe
  • Pipelines
  • Notebook
  • Manifests

[annajung]

Discussion around dependency versions for 1.6 https://github.com/kubeflow/manifests/issues/2207

[Tomcli]

For kfp-tekton project, we want to use kfp-tekton v1.2.1 release for KF v1.6. The update script should be available at https://github.com/kubeflow/manifests/blob/master/hack/sync-kfp-tekton-manifests.sh. Thanks.

[annajung]

Due to a blocker that was identified last week, we are delaying the release for 2 weeks, with the possibility of another extension. The new feature freeze date is Wednesday, Jun 15th, 2022.

• Tracking issue: https://github.com/kubeflow/testing/issues/1006
• Email announcement: https://groups.google.com/u/2/g/kubeflow-discuss
• New schedule PR: https://github.com/kubeflow/community/pull/561

[annajung]

Meeting notes from June 13th release team meeting:

• Dependencies for 1.6

  • Kubernetes 1.22
  • Istio 1.13
  • Cert Manager v1.5.0
  • Knative 1.4
  • Kustomize version (Need to confirm with manifest WG)

• WG branch/tags used for Kubeflow 1.6

  • KFP tekton v1.2.1
  • KFP backend 2.0.0-alpha (Not available yet)
  • Notebook v1.6.0 (Not available yet)
  • KServe 0.9 if available, if not 0.8
  • Katib v0.14.0 (Not available yet)
  • Training-Operators v1.5.0 (Not available yet)

cc @kubeflow/wg-manifests-leads @kimwnasptd

[annajung]

After checking with all WGs, the release team is delaying the release for 2 more weeks. The new feature freeze date is Wednesday, Jun 29th, 2022.

• Email announcement: https://groups.google.com/g/kubeflow-discuss/c/I4l97HvrGEA/m/227aCe_mCgAJ
• New schedule PR: https://github.com/kubeflow/community/pull/562

[annajung]

Kubeflow 1.6.0-rc.0 is now available!

• release tag: https://github.com/kubeflow/manifests/releases/tag/v1.6.0-rc.0
• announcement: https://groups.google.com/g/kubeflow-discuss/c/wGZJUpkVVBs/m/-ucjru9RAAAJ

[kimwnasptd]

Small heads up that there's a confirmed bug with the latest Jupyter web app included in RC0 https://github.com/kubeflow/kubeflow/issues/6572.

This was because I ended up pushing a locally built image that used incorrect cache. This will be resolved on the next RC, once we'll have rebuilt the images of kubeflow/kubeflow

[juliusvonkohout]

Small heads up that there's a confirmed bug with the latest Jupyter web app included in RC0 kubeflow/kubeflow#6572.

This was because I ended up pushing a locally built image that used incorrect cache. This will be resolved on the next RC, once we'll have rebuilt the images of kubeflow/kubeflow

I also left a typo in the networkpolicies for 1.6 https://github.com/kubeflow/manifests/pull/2254 . By the way kubeflow/manifests is missing the stalebot that other kubeflow repositories have https://github.com/apps/stale .

[jhasm]

Small heads up that there's a confirmed bug with the latest Jupyter web app included in RC0 kubeflow/kubeflow#6572.

This was because I ended up pushing a locally built image that used incorrect cache. This will be resolved on the next RC, once we'll have rebuilt the images of kubeflow/kubeflow

Hi @kimwnasptd when should we expect the next RC with the Jupyter web app bug fixed? And in the meantime, what are the instructions for folks to work around this?

I see comments on https://github.com/kubeflow/kubeflow/issues/6572 about building from source. Can you please elaborate on this and point to some instructions? What should we build from source and how?

[annajung]

A release blocking issue has been identified and needs extra time to resolve the issue. Therefore, the release has been delayed. The new proposed release date is Wednesday, Sep 7th 2022 and seeking feedback from the community

Email announcement: https://groups.google.com/g/kubeflow-discuss/c/h0NX0XyUo74/m/6GTCsD7KAAAJ New proposed release timeline PR: https://github.com/kubeflow/community/pull/569

[annajung]

Kubeflow RC.3 is available

announcement: https://groups.google.com/u/2/g/kubeflow-discuss/c/Kwb6qO2_Hf8

[juliusvonkohout]

I have some proposition for 1.6.1 already. Something went wrong with the notebook culling https://github.com/kubeflow/kubeflow/issues/6648

[juliusvonkohout]

and another big regression in 1.6 https://github.com/kubeflow/pipelines/issues/8256

[annajung]

Thanks for identifying these issues @juliusvonkohout.

@kimwnasptd @kubeflow/wg-notebooks-leads @zijianjoy @kubeflow/wg-pipeline-leads, let's continue using this issue to track progress for 1.6.1.

[nithin8702]

Hi @annajung Seldon is not working due to k8s migration 1.22. Shall we also include this?

[annajung]

@nithin8702 could you tag the specific issue you're referring to?

[nithin8702]

https://github.com/kubeflow/manifests/issues/2207#issuecomment-1245664517

[surajkota]

@annajung @kimwnasptd Created istio issue as discussed https://github.com/kubeflow/manifests/issues/2285

[annajung]

Hi everyone, 1.6.1-rc.0 is now available which includes fixes for

• Notebook controller constantly culling every notebook
• Pipelines "Unknown execution spec" error
• Manifest Istio configuration patch

kubeflow-discuss announcement: https://groups.google.com/g/kubeflow-discuss/c/-0fhFVnj1j8/m/XFVGAzxjBAAJ

[surajkota]

@annajung @kimwnasptd @zijianjoy Kubeflow pipelines has been updated from 2.0.0-alpha.3 to 2.0.0-alpha.5 in 1.6.1-rc0 and there are several new features in it https://github.com/kubeflow/pipelines/blob/2.0.0-alpha.5/CHANGELOG.md. Patch releases should ideally only include bug fixes. We/AWS would like to have some more time to test it and we should also set the right expectations for the customers.

[juliusvonkohout]

@surajkota. Yes because KFP is in a transition phase. As soon as 2.0 is released that will not happen anymore. And trust me you want this version since it includes significant security fixes from me and @difince.

[annajung]

Hi folks, @surajkota brought up a really good point about pipelines alpha.5, therefore, I want to give the community a few more days to test against Kubeflow 1.6.1-rc.0 before cutting the final release.

The release team previously mentioned that we'll have a final cut available today Oct 3rd, but we will delay cutting the final release until this next Monday, October 10th.

Please bring up any issues identified with the 1.6.1 RC.0 in this issue

[annajung]

https://github.com/kubeflow/kubeflow/pull/6664 was brought up in kubeflow-discuss mailing group tagging @kimwnasptd @kubeflow/wg-notebooks-leads for their review

[kimwnasptd]

I'd like to avoid adding any further fixes for 1.6.1, that are not critical for security or render the platform unusable. So I'd propose to finalize the RC and deliver the release on Monday.

With that said though, we'll review https://github.com/kubeflow/kubeflow/pull/6664 to ensure it'll be part of the next release

[annajung]

Hi everyone, 1.6.1 is now available

kubeflow-discuss announcement: https://groups.google.com/g/kubeflow-discuss/c/amsxyXbY_nk/m/FaWxOd4VBAAJ

[juliusvonkohout]

The istio nodeport (loadbalancer)instead of a proper ClusterIP also seems dangerous to me https://oteemo.com/think-nodeport-kubernetes/

I used the following to fix it https://github.com/kubeflow/manifests/pull/2296

[annajung]

With 1.6.1 released and 1.7 release started, closing out the issue.