Feature: Kubernetes Addon Selection and Defaults Override

[bernardhalas]

Motivation

Claudie intends to be a fully usable Kubernetes distribution with little setup effort. For that reason, it needs to come pre-packaged with storage system (Longhorn) or observability stack (Prometheus, Alertmanager, Grafana,...). However, the defaults, that these addons come with do not suit all users and they wish to have a possibility to override them.

Description

• Design a way how to opt-out from addons.
• Design a way how to override defaults for the addons.

Exit criteria

• [ ] Users are able to opt-in/opt-out from the addons
• [ ] Users are able to override default addons deployment manifests.

[bernardhalas]

The proposal is to utilize Sveltos for templating Kubernetes manifests. Sveltos can load values from both management and managed clusters, from ConfigMaps and from Secrets. This would mean that Claudie will be deployed together with Sveltos manifests on the management clusters, and Claudie will create the necessary ConfigMaps and Secrets, for Sveltos to consume.

Now, for customizing Kubernetes manifests the users are recommended to use kustomize. This would mean, that Claudie will provide manifest bases and users will just supply the overlay for those bases.

Tthere will be two repositories to come into play:

1. berops/claudie-kustomize-bases for providing default Kustomize bases for Claudie addons; with a very rare need for forking
2. berops/claudie-templates for providing overlays, patches and opt-ins/opt-outs for for Claudie addons; expected to be overriden by users for modifying the default Claudie behavior.

e.g.:

https://github.com/berops/claudie-templates/addons/kustomize.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- https://github.com/berops/claudie-kustomize-bases//longhorn?ref=v0.9.10
- https://github.com/berops/claudie-kustomize-bases//prometheus?ref=v0.9.10

patches:
- patches/nvme-storage-class.yml
- patches/observability-taints.yml

Related to #1758 .