Upgrades in 1.1 should follow kustomize off the shelf workflow

[jlewi]

trafficstars

Filing this issue to track simplifying the upgrade process in Kubeflow 1.1.

Here's the current instructions for how Kubeflow upgrades are done. https://www.kubeflow.org/docs/upgrading/upgrade/

This differs from the standard off the shelf workflow for kustomize applications https://github.com/kubernetes-sigs/kustomize/blob/master/docs/workflows.md#off-the-shelf-configuration

In particular, we introduce a KFUpgrade resource which defines pointers to the old and new KFDef. https://www.kubeflow.org/docs/upgrading/upgrade/#upgrade-instructions

kfctl then does a lot of a magic in order to try to reapply any user defined kustomizations ontop of the new configs.

With the new kustomize patterns (http://bit.ly/kf_kustomize_v3) we should be able to simplify this and I think eliminate the need for kfctl. Instead users should be able to just

1. Update .cache to point to a new version of the kubeflow/manifests directory
2. Run kustomize build to regenerate the package.

This is because the new pattern with stacks is that kfctl generates a new kustomize package using Kubeflow defined packages in .cache as the base. So a user can regenerate .cache without losing any of their kustomizations.

There are a couple of issues that we run into when applying the updated manifests

1. Pruning - how do we cleanup resources from earlier versions of Kubeflow that are no longer in the latest Kubeflow resource
2. Updating immutable fields - Certain fields are immutable and will cause errors when apply is called.

Rather than rely on kfctl logic to solve these problems we should follow a shift left pattern. Our expectation should be that we rely on existing tools (e.g. kubectl, kpt, etc...) to apply the manifests and handle these problems.

kpt for example supports pruning

/cc @richardsliu @yanniszark @kunmingg

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

Label	Probability
kind/feature	0.72

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.

[jlewi]

As noted in kubeflow/kubeflow#4873; kustomize commonLabels should only be used for immutable labels https://kubectl.docs.kubernetes.io/pages/app_management/labels_and_annotations.html

As these are used in selectors which are immutable.

Right now our applications are using version in the version and instance label which is used in selector and set via commonLabels. https://github.com/kubeflow/manifests/blob/abc6898ba535515e88846e7cc97faa208ffdacb9/jupyter/jupyter-web-app/overlays/application/kustomization.yaml#L11

We need to fix this so that labels will be immutable across version updates.

It looks like kubectl has alpha support in prune.

  # Note: --prune is still in Alpha
  # Apply the configuration in manifest.yaml that matches label app=nginx and delete all the other resources that are
not in the file and match label app=nginx.
  kubectl apply --prune -f manifest.yaml -l app=nginx

So if we have appropriate, immutable, labels for each application then we should be able to use kubectl to apply a new version and prune any removed resources.

[jbottum]

@jlewi Hey Jeremy - will this feature be included in Kubeflow 1.1 ?