pipelines chore(manifests): refactor manifests for kustomize5 compatibility. Part of #10053

Description of your changes:

This PR is part of larger initiative https://github.com/kubeflow/pipelines/issues/10053 to refactor manifests and addresses the problems of kustomize5 compatibility.

Warnings thrown before this PR:

$ kustomize build apps/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user/  

# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
2023/10/12 15:10:41 well-defined vars that were never replaced: kfp-app-name,kfp-app-version

Expected behaviour after refactoring the resources for kustomize5 is to obtain healthy set of manifests without any warnings.

Remaining Warnings (Work in progress):

 Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
2023/10/12 15:50:46 well-defined vars that were never replaced: kfp-app-version,kfp-app-name

Changes

This PR should fix the deprecated fields for the ml pipelines kustomization stack. The deprecated fields in kustomize5 to refactor:

bases -> resources
patchesStrategicMerge -> patches
patchesJson6902 -> patchces
vars -> replacements

After couple more commits I am expecting to have also refactored the remaining vars and patchesJson6902 deprecations from the rest of the resources.

Looking for feedback! (@annajung, @midhun1998)

Checklist:

[x] The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

Oct 12 '23 13:10 rawc0der

Hi @rawc0der. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Oct 12 '23 13:10 google-oss-prow[bot]

/ok-to-test

Oct 12 '23 22:10 zijianjoy

@rawc0der it is usually very helpful to join the kfp working group meeting to speed up PR review and discuss open issues and PRs

Oct 13 '23 17:10 juliusvonkohout

/retest

Jan 12 '24 12:01 juliusvonkohout

@zijianjoy this is definitely something that we want for Kubeflow 1.9 in the manifests WG.

@rawc0der please fix the integration tests

Jan 12 '24 12:01 juliusvonkohout

@rawc0der please fix the integration tests

Great, then I'll find the time to do it this week!

Jan 23 '24 08:01 rawc0der

/retest

Jan 25 '24 11:01 rawc0der

Integration tests have been fixed. Some changes were required in the install scripts where older version of Kustomize v2.0.9 was used. Otherwise there are no more patchesJson6902, patchesStrategicMerge or bases references.

More work is needed for replacing the remaining vars. Working on this and should be ready soon.

FYI @juliusvonkohout

Jan 25 '24 16:01 rawc0der

@rawc0der please join the KFP meeting on January 31 and put your PR on the agenda in the document in advance. This way we might get this merged soon.

Jan 26 '24 12:01 juliusvonkohout

@rawc0der please join the KFP meeting on January 31 and put your PR on the agenda in the document in advance. This way we might get this merged soon.

Sure, I will try to join the next KFP meeting to move this forward.

Jan 26 '24 12:01 rawc0der

Thank you for the contributions! Would you like to also update other places in the repo? https://github.com/kubeflow/pipelines/blob/05f69b233378e1b0351bf40ab037830f53738b15/manifests/kustomize/hack/presubmit.sh#L26 (Just to make sure this doesn't break our postsubmit test)

Feb 02 '24 00:02 zijianjoy

Sure, I can do that. Any other pointers?

Feb 02 '24 00:02 rawc0der

The testing mechanism is defined in this folder: https://github.com/GoogleCloudPlatform/oss-test-infra/tree/3f8905d98cdbb7005887113ae692b4024b83188f/prow/prowjobs/kubeflow/pipelines, you can take a look and see if anything needs to be updated.

Feb 02 '24 21:02 zijianjoy

@rawc0der Did you have the chance to evaluate the tests?

@zijianjoy Is there a way to run the postsubmit test in a local environment?

Mar 06 '24 18:03 rimolive

I will try to spend some time today to wrap this up.

Mar 07 '24 10:03 rawc0der

@rawc0der Any updates? Is there something I can help you?

Mar 11 '24 18:03 rimolive

Hi @rimolive, I've run the test locally. No problems when building the templates. Here's the output.

(⎈ |kind-k8s-1.26:default)$ bash test.sh  

+++ dirname test.sh
++ cd .
++ pwd
+ DIR=/Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack
+ MANIFESTS_DIR=/Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/..
+ kubectl version --client=true
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.7
+ kustomize version
v5.0.3
+ kpt version
1.0.0-beta.49
+ kustomization_yamls=("cluster-scoped-resources" "base/installs/generic" "env/dev" "env/gcp" "env/platform-agnostic" "env/platform-agnostic-pns" "env/platform-agnostic-emissary" "env/aws" "env/azure")
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../cluster-scoped-resources
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../base/installs/generic
I0312 23:00:59.093164   18920 log.go:198] well-defined vars that were never replaced: kfp-namespace,kfp-app-name,kfp-app-version,kfp-artifact-bucket-name
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/dev
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/gcp
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/platform-agnostic
I0312 23:00:59.997132   18925 log.go:198] well-defined vars that were never replaced: kfp-app-name,kfp-app-version
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/platform-agnostic-pns
I0312 23:01:00.301708   18926 log.go:198] well-defined vars that were never replaced: kfp-app-name,kfp-app-version
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/platform-agnostic-emissary
I0312 23:01:00.590625   18928 log.go:198] well-defined vars that were never replaced: kfp-app-name,kfp-app-version
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/aws
I0312 23:01:00.915687   18929 log.go:198] well-defined vars that were never replaced: kfp-namespace,kfp-app-name,kfp-app-version
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/azure
I0312 23:01:01.184173   18930 log.go:198] well-defined vars that were never replaced: kfp-app-version,kfp-app-name
+ kustomization_yamls_v3=("base/installs/multi-user" "env/platform-agnostic-multi-user" "env/platform-agnostic-multi-user-pns" "env/platform-agnostic-multi-user-emissary")
+ for path in '"${kustomization_yamls_v3[@]}"'
+ kustomize build /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../base/installs/multi-user
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
2024/03/12 23:01:01 well-defined vars that were never replaced: kfp-app-name,kfp-app-version,kfp-artifact-bucket-name
+ for path in '"${kustomization_yamls_v3[@]}"'
+ kustomize build /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/platform-agnostic-multi-user
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
2024/03/12 23:01:01 well-defined vars that were never replaced: kfp-app-name,kfp-app-version
+ for path in '"${kustomization_yamls_v3[@]}"'
+ kustomize build /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/platform-agnostic-multi-user-pns
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
2024/03/12 23:01:02 well-defined vars that were never replaced: kfp-app-name,kfp-app-version
+ for path in '"${kustomization_yamls_v3[@]}"'
+ kustomize build /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/platform-agnostic-multi-user-emissary
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
2024/03/12 23:01:02 well-defined vars that were never replaced: kfp-app-name,kfp-app-version
+ kpt pkg tree /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/..

(⎈ |kind-k8s-1.26:default)$ echo $?                
0

kpt resources output

``` $ kpt pkg tree ../ (⎈ |kind-k8s-1.26:default) .. ├── application │ ├── [application.yaml] Application $(kfp-app-name) │ └── [kustomization.yaml] Kustomization ├── cache │ ├── [cache-deployment.yaml] Deployment cache-server │ ├── [cache-role.yaml] Role kubeflow-pipelines-cache-role │ ├── [cache-rolebinding.yaml] RoleBinding kubeflow-pipelines-cache-binding │ ├── [cache-sa.yaml] ServiceAccount kubeflow-pipelines-cache │ ├── [cache-service.yaml] Service cache-server │ ├── [kustomization.yaml] Kustomization │ └── cache-deployer │ ├── [cache-deployer-deployment.yaml] Deployment cache-deployer-deployment │ ├── [cache-deployer-role.yaml] Role kubeflow-pipelines-cache-deployer-role │ ├── [cache-deployer-rolebinding.yaml] RoleBinding kubeflow-pipelines-cache-deployer-rolebinding │ ├── [kustomization.yaml] Kustomization │ └── cluster-scoped │ ├── [cache-deployer-clusterrole.yaml] ClusterRole kubeflow-pipelines-cache-deployer-clusterrole │ ├── [cache-deployer-clusterrolebinding.yaml] ClusterRoleBinding kubeflow-pipelines-cache-deployer-clusterrolebinding │ ├── [cache-deployer-sa.yaml] ServiceAccount kubeflow-pipelines-cache-deployer-sa │ └── [kustomization.yaml] Kustomization ├── generic │ ├── [kustomization.yaml] Kustomization │ ├── [mysql-secret.yaml] Secret mysql-secret │ ├── [pipeline-install-config.yaml] ConfigMap pipeline-install-config │ └── postgres │ ├── [kustomization.yaml] Kustomization │ ├── [pipeline-install-config.yaml] ConfigMap pipeline-install-config │ └── [postgres-secret-extended.yaml] Secret postgres-secret-extended ├── multi-user │ ├── [istio-authorization-config.yaml] DestinationRule ml-pipeline │ ├── [istio-authorization-config.yaml] DestinationRule ml-pipeline-ui │ ├── [istio-authorization-config.yaml] DestinationRule ml-pipeline-visualizationserver │ ├── [istio-authorization-config.yaml] AuthorizationPolicy kubeflow/ml-pipeline │ ├── [istio-authorization-config.yaml] AuthorizationPolicy kubeflow/ml-pipeline-ui │ ├── [istio-authorization-config.yaml] AuthorizationPolicy kubeflow/ml-pipeline-visualizationserver │ ├── [istio-authorization-config.yaml] AuthorizationPolicy kubeflow/service-cache-server │ ├── [kustomization.yaml] Kustomization │ ├── [view-edit-cluster-roles.yaml] ClusterRole aggregate-to-kubeflow-pipelines-edit │ ├── [view-edit-cluster-roles.yaml] ClusterRole aggregate-to-kubeflow-pipelines-view │ ├── [view-edit-cluster-roles.yaml] ClusterRole kubeflow-pipelines-edit │ ├── [view-edit-cluster-roles.yaml] ClusterRole kubeflow-pipelines-view │ ├── [virtual-service.yaml] VirtualService ml-pipeline-ui │ ├── api-service │ │ ├── [cluster-role-binding.yaml] ClusterRoleBinding ml-pipeline │ │ ├── [cluster-role.yaml] ClusterRole ml-pipeline │ │ ├── [deployment-patch.yaml] Deployment ml-pipeline │ │ └── [kustomization.yaml] Kustomization │ ├── cache │ │ ├── [cluster-role-binding.yaml] ClusterRoleBinding kubeflow-pipelines-cache-binding │ │ ├── [cluster-role.yaml] ClusterRole kubeflow-pipelines-cache-role │ │ ├── [deployment-patch.yaml] Deployment cache-server │ │ └── [kustomization.yaml] Kustomization │ ├── metadata-writer │ │ ├── [cluster-role-binding.yaml] ClusterRoleBinding kubeflow-pipelines-metadata-writer-binding │ │ ├── [cluster-role.yaml] ClusterRole kubeflow-pipelines-metadata-writer-role │ │ ├── [deployment-patch.yaml] Deployment metadata-writer │ │ └── [kustomization.yaml] Kustomization │ ├── persistence-agent │ │ ├── [cluster-role-binding.yaml] ClusterRoleBinding ml-pipeline-persistenceagent-binding │ │ ├── [cluster-role.yaml] ClusterRole ml-pipeline-persistenceagent-role │ │ ├── [deployment-patch.yaml] Deployment ml-pipeline-persistenceagent │ │ └── [kustomization.yaml] Kustomization │ ├── pipelines-profile-controller │ │ ├── [composite-controller.yaml] CompositeController kubeflow-pipelines-profile-controller │ │ ├── [deployment.yaml] Deployment kubeflow-pipelines-profile-controller │ │ ├── [kustomization.yaml] Kustomization │ │ └── [service.yaml] Service kubeflow-pipelines-profile-controller │ ├── pipelines-ui │ │ ├── [cluster-role-binding.yaml] ClusterRoleBinding ml-pipeline-ui │ │ ├── [cluster-role.yaml] ClusterRole ml-pipeline-ui │ │ ├── [configmap-patch.yaml] ConfigMap ml-pipeline-ui-configmap │ │ ├── [deployment-patch.yaml] Deployment ml-pipeline-ui │ │ └── [kustomization.yaml] Kustomization │ ├── scheduled-workflow │ │ ├── [cluster-role-binding.yaml] ClusterRoleBinding ml-pipeline-scheduledworkflow-binding │ │ ├── [cluster-role.yaml] ClusterRole ml-pipeline-scheduledworkflow-role │ │ ├── [deployment-patch.yaml] Deployment ml-pipeline-scheduledworkflow │ │ └── [kustomization.yaml] Kustomization │ └── viewer-controller │ ├── [cluster-role-binding.yaml] ClusterRoleBinding ml-pipeline-viewer-crd-binding │ ├── [cluster-role.yaml] ClusterRole ml-pipeline-viewer-controller-role │ ├── [deployment-patch.yaml] Deployment ml-pipeline-viewer-crd │ └── [kustomization.yaml] Kustomization ├── base │ ├── [kustomization.yaml] Kustomization │ ├── [metadata-envoy-deployment.yaml] Deployment metadata-envoy-deployment │ ├── [metadata-envoy-service.yaml] Service metadata-envoy-service │ ├── [metadata-grpc-configmap.yaml] ConfigMap metadata-grpc-configmap │ ├── [metadata-grpc-deployment.yaml] Deployment metadata-grpc-deployment │ ├── [metadata-grpc-sa.yaml] ServiceAccount metadata-grpc-server │ └── [metadata-grpc-service.yaml] Service metadata-grpc-service ├── istio │ ├── [destination-rule.yaml] DestinationRule metadata-grpc-service │ ├── [istio-authorization-policy.yaml] AuthorizationPolicy metadata-grpc-service │ ├── [kustomization.yaml] Kustomization │ └── [virtual-service.yaml] VirtualService kubeflow/metadata-grpc ├── db │ ├── [kustomization.yaml] Kustomization │ ├── [metadata-db-deployment.yaml] Deployment metadata-db │ ├── [metadata-db-pvc.yaml] PersistentVolumeClaim metadata-mysql │ ├── [metadata-db-service.yaml] Service metadata-db │ └── patches │ └── [metadata-grpc-deployment.yaml] Deployment metadata-grpc-deployment ├── postgres │ ├── [kustomization.yaml] Kustomization │ ├── [metadata-db-deployment.yaml] Deployment metadata-postgres-db │ ├── [metadata-db-pvc.yaml] PersistentVolumeClaim metadata-postgres │ ├── [metadata-db-service.yaml] Service metadata-postgres-db │ └── patches │ └── [metadata-grpc-deployment.yaml] Deployment metadata-grpc-deployment ├── pipeline │ ├── [container-builder-sa.yaml] ServiceAccount kubeflow-pipelines-container-builder │ ├── [kfp-launcher-configmap.yaml] ConfigMap kfp-launcher │ ├── [kustomization.yaml] Kustomization │ ├── [ml-pipeline-apiserver-deployment.yaml] Deployment ml-pipeline │ ├── [ml-pipeline-apiserver-role.yaml] Role ml-pipeline │ ├── [ml-pipeline-apiserver-rolebinding.yaml] RoleBinding ml-pipeline │ ├── [ml-pipeline-apiserver-sa.yaml] ServiceAccount ml-pipeline │ ├── [ml-pipeline-apiserver-service.yaml] Service ml-pipeline │ ├── [ml-pipeline-persistenceagent-deployment.yaml] Deployment ml-pipeline-persistenceagent │ ├── [ml-pipeline-persistenceagent-role.yaml] Role ml-pipeline-persistenceagent-role │ ├── [ml-pipeline-persistenceagent-rolebinding.yaml] RoleBinding ml-pipeline-persistenceagent-binding │ ├── [ml-pipeline-persistenceagent-sa.yaml] ServiceAccount ml-pipeline-persistenceagent │ ├── [ml-pipeline-scheduledworkflow-deployment.yaml] Deployment ml-pipeline-scheduledworkflow │ ├── [ml-pipeline-scheduledworkflow-role.yaml] Role ml-pipeline-scheduledworkflow-role │ ├── [ml-pipeline-scheduledworkflow-rolebinding.yaml] RoleBinding ml-pipeline-scheduledworkflow-binding │ ├── [ml-pipeline-scheduledworkflow-sa.yaml] ServiceAccount ml-pipeline-scheduledworkflow │ ├── [ml-pipeline-ui-configmap.yaml] ConfigMap ml-pipeline-ui-configmap │ ├── [ml-pipeline-ui-deployment.yaml] Deployment ml-pipeline-ui │ ├── [ml-pipeline-ui-role.yaml] Role ml-pipeline-ui │ ├── [ml-pipeline-ui-rolebinding.yaml] RoleBinding ml-pipeline-ui │ ├── [ml-pipeline-ui-sa.yaml] ServiceAccount ml-pipeline-ui │ ├── [ml-pipeline-ui-service.yaml] Service ml-pipeline-ui │ ├── [ml-pipeline-viewer-crd-deployment.yaml] Deployment ml-pipeline-viewer-crd │ ├── [ml-pipeline-viewer-crd-role.yaml] Role ml-pipeline-viewer-controller-role │ ├── [ml-pipeline-viewer-crd-rolebinding.yaml] RoleBinding ml-pipeline-viewer-crd-binding │ ├── [ml-pipeline-viewer-crd-sa.yaml] ServiceAccount ml-pipeline-viewer-crd-service-account │ ├── [ml-pipeline-visualization-deployment.yaml] Deployment ml-pipeline-visualizationserver │ ├── [ml-pipeline-visualization-sa.yaml] ServiceAccount ml-pipeline-visualizationserver │ ├── [ml-pipeline-visualization-service.yaml] Service ml-pipeline-visualizationserver │ ├── [pipeline-runner-role.yaml] Role pipeline-runner │ ├── [pipeline-runner-rolebinding.yaml] RoleBinding pipeline-runner-binding │ ├── [pipeline-runner-sa.yaml] ServiceAccount pipeline-runner │ ├── [viewer-sa.yaml] ServiceAccount kubeflow-pipelines-viewer │ ├── cluster-scoped │ │ ├── [kustomization.yaml] Kustomization │ │ ├── [scheduled-workflow-crd.yaml] CustomResourceDefinition scheduledworkflows.kubeflow.org │ │ └── [viewer-crd.yaml] CustomResourceDefinition viewers.kubeflow.org │ └── metadata-writer │ ├── [kustomization.yaml] Kustomization │ ├── [metadata-writer-deployment.yaml] Deployment metadata-writer │ ├── [metadata-writer-role.yaml] Role kubeflow-pipelines-metadata-writer-role │ ├── [metadata-writer-rolebinding.yaml] RoleBinding kubeflow-pipelines-metadata-writer-binding │ └── [metadata-writer-sa.yaml] ServiceAccount kubeflow-pipelines-metadata-writer ├── cache │ ├── [cache-deployment-patch.yaml] Deployment cache-server │ └── [kustomization.yaml] Kustomization ├── pipeline │ ├── [kustomization.yaml] Kustomization │ └── [ml-pipeline-apiserver-deployment-patch.yaml] Deployment ml-pipeline ├── cluster-scoped-resources │ ├── [kustomization.yaml] Kustomization │ └── [namespace.yaml] Namespace $(kfp-cluster-scoped-namespace) ├── aws │ ├── [aws-configuration-pipeline-patch.yaml] Deployment ml-pipeline │ ├── [aws-configuration-pipeline-ui-patch.yaml] Deployment ml-pipeline-ui │ └── [kustomization.yaml] Kustomization ├── azure │ ├── [kustomization.yaml] Kustomization │ └── minio-azure-gateway │ ├── [kustomization.yaml] Kustomization │ ├── [minio-azure-gateway-deployment.yaml] Deployment minio │ └── [minio-azure-gateway-service.yaml] Service minio-service ├── base │ ├── [cache-cert-issuer.yaml] Issuer kfp-cache-selfsigned-issuer │ ├── [cache-cert.yaml] Certificate kfp-cache-cert │ ├── [cache-webhook-config.yaml] MutatingWebhookConfiguration cache-webhook-kubeflow │ └── [kustomization.yaml] Kustomization ├── cluster-scoped-resources │ └── [kustomization.yaml] Kustomization ├── dev │ ├── [delete-cache-deployer.yaml] Deployment cache-deployer-deployment │ ├── [delete-cache-deployer.yaml] Role kubeflow-pipelines-cache-deployer-role │ ├── [delete-cache-deployer.yaml] RoleBinding kubeflow-pipelines-cache-deployer-rolebinding │ ├── [kustomization.yaml] Kustomization │ └── [namespace.yaml] Namespace $(kfp-dev-namespace) ├── platform-agnostic-multi-user │ ├── [kustomization.yaml] Kustomization │ └── patches │ ├── [delete.clusterrole.cache-deployer.yaml] ClusterRole kubeflow-pipelines-cache-deployer-clusterrole │ ├── [delete.crb.cache-deployer.yaml] ClusterRoleBinding kubeflow-pipelines-cache-deployer-clusterrolebinding │ ├── [delete.deployment.cache-deployer.yaml] Deployment cache-deployer-deployment │ ├── [delete.role.cache-deployer.yaml] Role kubeflow-pipelines-cache-deployer-role │ ├── [delete.rolebinding.cache-deployer.yaml] RoleBinding kubeflow-pipelines-cache-deployer-rolebinding │ └── [delete.sa.cache-deployer.yaml] ServiceAccount kubeflow-pipelines-cache-deployer-sa ├── dev │ ├── [kustomization.yaml] Kustomization │ └── postgresql │ └── [kustomization.yaml] Kustomization ├── gcp │ ├── [gcp-configurations-patch.yaml] Deployment ml-pipeline │ ├── [kustomization.yaml] Kustomization │ ├── cloudsql-proxy │ │ ├── [cloudsql-proxy-deployment.yaml] Deployment cloudsqlproxy │ │ ├── [cloudsql-proxy-sa.yaml] ServiceAccount kubeflow-pipelines-cloudsql-proxy │ │ ├── [kustomization.yaml] Kustomization │ │ └── [mysql-service.yaml] Service mysql │ ├── inverse-proxy │ │ ├── [kustomization.yaml] Kustomization │ │ ├── [proxy-configmap.yaml] ConfigMap inverse-proxy-config │ │ ├── [proxy-deployment.yaml] Deployment proxy-agent │ │ ├── [proxy-role.yaml] Role proxy-agent-runner │ │ ├── [proxy-rolebinding.yaml] RoleBinding proxy-agent-runner │ │ └── [proxy-sa.yaml] ServiceAccount proxy-agent-runner │ └── minio-gcs-gateway │ ├── [kustomization.yaml] Kustomization │ ├── [minio-gcs-gateway-deployment.yaml] Deployment minio │ ├── [minio-gcs-gateway-sa.yaml] ServiceAccount kubeflow-pipelines-minio-gcs-gateway │ └── [minio-gcs-gateway-service.yaml] Service minio-service ├── platform-agnostic │ ├── [kustomization.yaml] Kustomization │ ├── platform-agnostic-emissary │ │ ├── [kustomization.yaml] Kustomization │ │ └── [workflow-controller-configmap-patch.yaml] ConfigMap workflow-controller-configmap │ ├── platform-agnostic-multi-user │ │ ├── [kustomization.yaml] Kustomization │ │ └── platform-agnostic-multi-user-pns │ │ ├── [kustomization.yaml] Kustomization │ │ └── [workflow-controller-configmap-patch.yaml] ConfigMap workflow-controller-configmap │ ├── platform-agnostic-multi-user-emissary │ │ ├── [kustomization.yaml] Kustomization │ │ └── [workflow-controller-configmap-patch.yaml] ConfigMap workflow-controller-configmap │ ├── platform-agnostic-multi-user-legacy │ │ └── [kustomization.yaml] Kustomization │ ├── platform-agnostic-pns │ │ ├── [kustomization.yaml] Kustomization │ │ └── [workflow-controller-configmap-patch.yaml] ConfigMap workflow-controller-configmap │ └── platform-agnostic-postgresql │ └── [kustomization.yaml] Kustomization ├── sample │ ├── [kustomization.yaml] Kustomization │ └── cluster-scoped-resources │ └── [kustomization.yaml] Kustomization ├── application │ ├── [application-controller-deployment.yaml] Deployment controller-manager │ ├── [application-controller-role.yaml] Role application-manager-role │ ├── [application-controller-rolebinding.yaml] RoleBinding application-manager-rolebinding │ ├── [application-controller-sa.yaml] ServiceAccount application │ ├── [application-controller-service.yaml] Service controller-manager-service │ ├── [kustomization.yaml] Kustomization │ └── cluster-scoped │ ├── [application-crd.yaml] CustomResourceDefinition applications.app.k8s.io │ └── [kustomization.yaml] Kustomization ├── Package "argo" │ ├── [Kptfile] Kptfile argo │ ├── base │ │ ├── [kustomization.yaml] Kustomization │ │ ├── [workflow-controller-configmap-patch.yaml] ConfigMap workflow-controller-configmap │ │ └── [workflow-controller-deployment-patch.yaml] Deployment workflow-controller │ ├── cluster │ │ └── [kustomization.yaml] Kustomization │ ├── namespace │ │ ├── [kustomization.yaml] Kustomization │ │ └── cluster-scoped │ │ └── [kustomization.yaml] Kustomization │ ├── Package "manifests" │ │ ├── [Kptfile] Kptfile manifests │ │ ├── base │ │ │ ├── [kustomization.yaml] Kustomization │ │ │ ├── full │ │ │ │ ├── [argoproj.io_clusterworkflowtemplates.yaml] CustomResourceDefinition clusterworkflowtemplates.argoproj.io │ │ │ │ ├── [argoproj.io_cronworkflows.yaml] CustomResourceDefinition cronworkflows.argoproj.io │ │ │ │ ├── [argoproj.io_workfloweventbindings.yaml] CustomResourceDefinition workfloweventbindings.argoproj.io │ │ │ │ ├── [argoproj.io_workflows.yaml] CustomResourceDefinition workflows.argoproj.io │ │ │ │ ├── [argoproj.io_workflowtaskresults.yaml] CustomResourceDefinition workflowtaskresults.argoproj.io │ │ │ │ ├── [argoproj.io_workflowtasksets.yaml] CustomResourceDefinition workflowtasksets.argoproj.io │ │ │ │ ├── [argoproj.io_workflowtemplates.yaml] CustomResourceDefinition workflowtemplates.argoproj.io │ │ │ │ └── [kustomization.yaml] Kustomization │ │ │ └── workflow-controller │ │ │ ├── [kustomization.yaml] Kustomization │ │ │ ├── [workflow-controller-configmap.yaml] ConfigMap workflow-controller-configmap │ │ │ ├── [workflow-controller-deployment.yaml] Deployment workflow-controller │ │ │ ├── [workflow-controller-metrics-service.yaml] Service workflow-controller-metrics │ │ │ ├── [workflow-controller-priorityclass.yaml] PriorityClass workflow-controller │ │ │ └── [workflow-controller-sa.yaml] ServiceAccount argo │ │ ├── cluster-install │ │ │ └── [kustomization.yaml] Kustomization │ │ ├── argo-server-rbac │ │ │ ├── [argo-server-clusterole.yaml] ClusterRole argo-server-cluster-role │ │ │ ├── [argo-server-clusterolebinding.yaml] ClusterRoleBinding argo-server-binding │ │ │ └── [kustomization.yaml] Kustomization │ │ ├── namespace-install │ │ │ └── [kustomization.yaml] Kustomization │ │ ├── argo-server-rbac │ │ │ ├── [argo-server-role.yaml] Role argo-server-role │ │ │ ├── [argo-server-rolebinding.yaml] RoleBinding argo-server-binding │ │ │ └── [kustomization.yaml] Kustomization │ │ ├── base │ │ │ ├── [agent-default-rolebinding.yaml] RoleBinding agent-default │ │ │ ├── [agent-role.yaml] Role agent │ │ │ ├── [argo-server-sso-secret.yaml] Secret argo-server-sso │ │ │ ├── [artifact-repositories-configmap.yaml] ConfigMap artifact-repositories │ │ │ ├── [cluster-workflow-template-rbac.yaml] ClusterRole argo-clusterworkflowtemplate-role │ │ │ ├── [cluster-workflow-template-rbac.yaml] ClusterRoleBinding argo-clusterworkflowtemplate-role-binding │ │ │ ├── [cluster-workflow-template-rbac.yaml] ClusterRole argo-server-clusterworkflowtemplate-role │ │ │ ├── [cluster-workflow-template-rbac.yaml] ClusterRoleBinding argo-server-clusterworkflowtemplate-role-binding │ │ │ ├── [executor-default-rolebinding.yaml] RoleBinding executor-default │ │ │ ├── [kustomization.yaml] Kustomization │ │ │ ├── [memoizer-default-rolebinding.yaml] RoleBinding memoizer-default │ │ │ ├── [memoizer-role.yaml] Role memoizer │ │ │ ├── [pod-manager-default-rolebinding.yaml] RoleBinding pod-manager-default │ │ │ ├── [pod-manager-role.yaml] Role pod-manager │ │ │ ├── [workflow-default-rolebinding.yaml] RoleBinding workflow-default-binding │ │ │ ├── [workflow-manager-default-rolebinding.yaml] RoleBinding workflow-manager-default │ │ │ ├── [workflow-manager-role.yaml] Role workflow-manager │ │ │ ├── emissary │ │ │ │ └── [executor-role.yaml] Role executor │ │ │ ├── pns │ │ │ │ └── [executor-role.yaml] Role executor │ │ │ ├── minio │ │ │ │ ├── [kustomization.yaml] Kustomization │ │ │ │ ├── [minio-deploy.yaml] Deployment minio │ │ │ │ ├── [minio-service.yaml] Service minio │ │ │ │ └── [my-minio-cred-secret.yaml] Secret my-minio-cred │ │ │ ├── overlays │ │ │ │ ├── [argo-server-deployment.yaml] Deployment argo-server │ │ │ │ └── [workflow-controller-configmap.yaml] ConfigMap workflow-controller-configmap │ │ │ ├── prometheus │ │ │ │ ├── [kustomization.yaml] Kustomization │ │ │ │ ├── [prometheus-config-cluster.yaml] ConfigMap prometheus-config │ │ │ │ ├── [prometheus-deployment.yaml] Deployment prometheus │ │ │ │ └── [prometheus-service.yaml] Service prometheus │ │ │ └── webhooks │ │ │ ├── [argo-workflows-webhook-clients-secret.yaml] Secret argo-workflows-webhook-clients │ │ │ ├── [github.com-rolebinding.yaml] RoleBinding github.com │ │ │ ├── [github.com-sa.yaml] ServiceAccount github.com │ │ │ ├── [kustomization.yaml] Kustomization │ │ │ └── [submit-workflow-template-role.yaml] Role submit-workflow-template │ │ ├── docker │ │ │ └── [executor-role.yaml] Role executor │ │ ├── k8sapi │ │ │ └── [executor-role.yaml] Role executor │ │ ├── kubelet │ │ │ ├── [executor-role.yaml] Role executor │ │ │ ├── [kubelet-executor-clusterrole.yaml] ClusterRole kubelet-executor │ │ │ └── [kubelet-executor-default-clusterrolebinding.yaml] ClusterRoleBinding kubelet-executor-default │ │ ├── minimal │ │ │ └── [kustomization.yaml] Kustomization │ │ ├── overlays │ │ │ └── [workflow-controller-configmap.yaml] ConfigMap workflow-controller-configmap │ │ ├── mysql │ │ │ ├── [argo-mysql-config-secret.yaml] Secret argo-mysql-config │ │ │ ├── [kustomization.yaml] Kustomization │ │ │ ├── [mysql-deployment.yaml] Deployment mysql │ │ │ ├── [mysql-service.yaml] Service mysql │ │ │ └── overlays │ │ │ └── [workflow-controller-configmap.yaml] ConfigMap workflow-controller-configmap │ │ ├── postgres │ │ │ ├── [argo-postgres-config-secret.yaml] Secret argo-postgres-config │ │ │ ├── [kustomization.yaml] Kustomization │ │ │ ├── [postgres-deployment.yaml] Deployment postgres │ │ │ ├── [postgres-service.yaml] Service postgres │ │ │ └── overlays │ │ │ └── [workflow-controller-configmap.yaml] ConfigMap workflow-controller-configmap │ │ └── dex │ │ ├── [dev-svc.yaml] Service dex │ │ ├── [dex-cm.yaml] ConfigMap dex │ │ ├── [dex-deploy.yaml] Deployment dex │ │ ├── [dex-rb.yaml] RoleBinding dex │ │ ├── [dex-role.yaml] Role dex │ │ ├── [dex-sa.yaml] ServiceAccount dex │ │ └── [kustomization.yaml] Kustomization │ ├── argo-server │ │ ├── [argo-server-deployment.yaml] Deployment argo-server │ │ ├── [argo-server-sa.yaml] ServiceAccount argo-server │ │ ├── [argo-server-service.yaml] Service argo-server │ │ └── [kustomization.yaml] Kustomization │ ├── crds │ │ ├── [kustomization.yaml] Kustomization │ │ └── minimal │ │ ├── [argoproj.io_clusterworkflowtemplates.yaml] CustomResourceDefinition clusterworkflowtemplates.argoproj.io │ │ ├── [argoproj.io_cronworkflows.yaml] CustomResourceDefinition cronworkflows.argoproj.io │ │ ├── [argoproj.io_workfloweventbindings.yaml] CustomResourceDefinition workfloweventbindings.argoproj.io │ │ ├── [argoproj.io_workflows.yaml] CustomResourceDefinition workflows.argoproj.io │ │ ├── [argoproj.io_workflowtaskresults.yaml] CustomResourceDefinition workflowtaskresults.argoproj.io │ │ ├── [argoproj.io_workflowtasksets.yaml] CustomResourceDefinition workflowtasksets.argoproj.io │ │ ├── [argoproj.io_workflowtemplates.yaml] CustomResourceDefinition workflowtemplates.argoproj.io │ │ └── [kustomization.yaml] Kustomization │ ├── workflow-controller-rbac │ │ ├── [kustomization.yaml] Kustomization │ │ ├── [workflow-aggregate-roles.yaml] ClusterRole argo-aggregate-to-admin │ │ ├── [workflow-aggregate-roles.yaml] ClusterRole argo-aggregate-to-edit │ │ ├── [workflow-aggregate-roles.yaml] ClusterRole argo-aggregate-to-view │ │ ├── [workflow-controller-clusterrole.yaml] ClusterRole argo-cluster-role │ │ ├── [workflow-controller-clusterrolebinding.yaml] ClusterRoleBinding argo-binding │ │ ├── [workflow-controller-role.yaml] Role argo-role │ │ └── [workflow-controller-rolebinding.yaml] RoleBinding argo-binding │ ├── workflow-controller-rbac │ │ ├── [kustomization.yaml] Kustomization │ │ ├── [workflow-controller-role.yaml] Role argo-role │ │ └── [workflow-controller-rolebinding.yaml] RoleBinding argo-binding │ └── sso │ ├── [kustomization.yaml] Kustomization │ └── overlays │ ├── [argo-server-sa.yaml] ServiceAccount argo-server │ └── [workflow-controller-configmap.yaml] ConfigMap workflow-controller-configmap ├── grafana │ ├── [grafana-deployment.yaml] Deployment kubeflow/grafana │ ├── [grafana-role.yaml] Role grafana │ ├── [grafana-rolebinding.yaml] RoleBinding grafana │ ├── [grafana-sa.yaml] ServiceAccount grafana │ ├── [grafana-service.yaml] Service grafana │ └── [kustomization.yaml] Kustomization ├── base │ ├── [cluster-role-binding.yaml] ClusterRoleBinding meta-controller-cluster-role-binding │ ├── [crd.yaml] CustomResourceDefinition compositecontrollers.metacontroller.k8s.io │ ├── [crd.yaml] CustomResourceDefinition controllerrevisions.metacontroller.k8s.io │ ├── [crd.yaml] CustomResourceDefinition decoratorcontrollers.metacontroller.k8s.io │ ├── [kustomization.yaml] Kustomization │ ├── [service-account.yaml] ServiceAccount meta-controller-service │ └── [stateful-set.yaml] StatefulSet metacontroller ├── base │ ├── [kustomization.yaml] Kustomization │ ├── [minio-deployment.yaml] Deployment minio │ ├── [minio-pvc.yaml] PersistentVolumeClaim minio-pvc │ ├── [minio-service.yaml] Service minio-service │ └── [mlpipeline-minio-artifact-secret.yaml] Secret mlpipeline-minio-artifact ├── istio │ ├── [istio-authorization-policy.yaml] AuthorizationPolicy minio-service │ ├── [istio-authorization-policy.yaml] DestinationRule ml-pipeline-minio │ └── [kustomization.yaml] Kustomization ├── base │ ├── [kustomization.yaml] Kustomization │ ├── [mysql-deployment.yaml] Deployment mysql │ ├── [mysql-pv-claim.yaml] PersistentVolumeClaim mysql-pv-claim │ ├── [mysql-service.yaml] Service mysql │ └── [mysql-serviceaccount.yaml] ServiceAccount mysql ├── istio │ ├── [istio-authorization-policy.yaml] DestinationRule ml-pipeline-mysql │ ├── [istio-authorization-policy.yaml] AuthorizationPolicy kubeflow/mysql │ └── [kustomization.yaml] Kustomization ├── base │ ├── [kustomization.yaml] Kustomization │ ├── [pg-deployment.yaml] Deployment postgres │ ├── [pg-pvc.yaml] PersistentVolumeClaim postgres-pvc │ ├── [pg-secret.yaml] Secret postgres-secret │ ├── [pg-service.yaml] Service postgres-service │ └── [pg-serviceaccount.yaml] ServiceAccount postgresql └── prometheus ├── [kustomization.yaml] Kustomization ├── [prometheus-configmap.yaml] ConfigMap prometheus-configmap ├── [prometheus-deployment.yaml] Deployment prometheus ├── [prometheus-role.yaml] Role prometheus ├── [prometheus-rolebinding.yaml] RoleBinding prometheus ├── [prometheus-sa.yaml] ServiceAccount prometheus └── [prometheus-service.yaml] Service prometheus ```

From my side think there are no more modifications (FYI vars are not modified anywhere for consistency).
I am waiting on some review and CI retest. What do you think?

Mar 12 '24 21:03 rawc0der

/retest

Mar 12 '24 21:03 rawc0der

Thank you, @rawc0der!

/lgtm

/assign @Tomcli @chensun @zijianjoy

Mar 12 '24 23:03 rimolive

We just switched to 5.2.1, but this should not block this PR https://github.com/kubeflow/manifests/pull/2654

Mar 15 '24 08:03 juliusvonkohout

I see, well I can update here as well to keep things aligned. Let me know.

Mar 15 '24 13:03 rawc0der

@rawc0der Yes, please. Make the changes to kustomize 5.2.1 and we'll be able to merge this PR.

Mar 15 '24 13:03 rimolive

@rawcoder you missed the number at the end of the string, where there is still a 5.0.3.

 ++ wget --no-verbose https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.2.1/kustomize_v5.0.3_linux_amd64.tar.gz -O kustomize_linux_amd64.tar.gz
https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv5.2.1/kustomize_v5.0.3_linux_amd64.tar.gz:
2024-03-17 08:41:14 ERROR 404: Not Found.

and you need to sign all of your commits with your github signing key.

Mar 17 '24 20:03 juliusvonkohout

/retest

Mar 18 '24 06:03 rawc0der

Looks good to me.

/lgtm

Mar 18 '24 15:03 rimolive

/lgtm

Mar 18 '24 17:03 juliusvonkohout

@zijianjoy Can you review/approve?

Mar 19 '24 22:03 rimolive

@Tomcli Can you review/approve?

Mar 20 '24 18:03 rimolive

Although this is probably safe, but let's wait until I cut a release first, so there's one less unknown (potential breaker).

/hold

Mar 20 '24 18:03 chensun

Undo LGTM.

Waiting for @Tomcli to add LGTM once his comment is addressed.

Mar 25 '24 20:03 chensun