[proposal] Support for installing Kubernetes apps using Omni

[smira]

Rationale

Omni allows to define cluster fully via the cluster templates, which allows to install machines, bring them into the cluster, assert they are ready and healthy. Cluster templates also allow to configure Talos Linux (and transitively, Kubernetes).

Sometimes there's an additional requirement to make cluster up and running, e.g.:

• install Cilium CNI (supported way is via Helm)
• bootstrap the initial Kubernetes app installer/updater (e.g. ArgoCD or Flux), which is also via Helm
• install some additional applications if the installation is simple, and gitops flow via ArgoCD/Flux is not desired

Today the only way to install Kubernetes apps is by using Talos machine config "extra bootstrap manifests" feature, but this feature is not based on helm, so the installed manifests are not tracked as installed by helm, and can't be easily managed later by helm. This adds extra bloat to the Talos machine configuration, which is not needed.

Omni can be in a perfect position to manage Kubernetes apps in the cluster: it is a single instance (vs. Talos controlplane machines which can be multiple for a cluster), it already has information about cluster health (knows when it's safe to install), it already has a language to describe the cluster (cluster templates).

Proposed Solution

As much as we are not happy with Helm, Helm is the de-facto standard.

For the initial phase, in order to simplify things, let's limit ourselves to the initial installation of Helm charts (skipping upgrades, changing chart values, etc.), as this is more simple, less risky, and solves the immediate problem of fully bootstrapping the cluster. In the future work, we might support updating charts as well.

As cluster templates are text YAML files, we should try to preserve this simple approach friendly to version control, expansion, templating, etc. The proposal is to use Helmfile as a language to describe what has to be installed.

We can add a field strategy and force it to be set to bootstrap-only to indicate that right now the charts are installed only once.

The initial scope is to support only charts available to Omni without auth or special setup, that is Omni should be able to download the charts from public repositories.

Cluster templates should sync the Helm instructions to an Omni resource (per cluster) describing charts to be installed.

Omni should have a controller which watches cluster status, and as soon as the cluster is ready (Kubernetes API is available), performs helm installation. Omni keeps the status of the install, and if the install was done, and strategy is bootstrap-only, it skips any work on this cluster/Helm chart.

Omni might keep a cache of downloaded Helm charts.

Future Work

• support updating/upgrading charts
• support private Helm charts
• support sharing Helmfile parts across clusters (i.e. enforcing a policy that e.g. cert-manager vX.Y should be installed for all clusters)
• showing pending updates/scheduling updates, etc.