Question about Scaling

[adberger]

Hi @stefanprodan

Thanks for the repo!

How does this multi-tenancy example scale in terms of tenants and namespaces? Imagine 3 Clusters with 10 Tenants and each of these Tenants have about 5 Namespaces (still a relatively small setup). That's a lot of YAML in my opinion. For each namespace you need your own GitRepository & Kustomization manifest. Of course you can generate most of the YAML with the flux-CLI, but mainting all this and having an clear overview is still hard.

Or am I missing something?

[kingdonb]

I think the intention is that you should find additional manifest generation tools to use with your environment.

There is work ongoing to show how this can be done with eg. GitHub Actions, if you are on GitHub, and for example Jsonnet if you want to use Jsonnet. The PR is nearing completion at fluxcd/flux2#1200.

Kustomize does so much and has a lot of tools that in my opinion Flux doesn't force you to learn all about, it is entirely possible to use Flux and never interact with Kustomize. It's a good tool for many uses, and a similar example is in fluxcd/flux2-kustomize-helm-example shows how it can be used to spread manifests across environments without duplicating YAML or making multiple copies.

There are many techniques that you can employ to help keep the cluster config small enough so that you can manage it all in your head, but Flux applies Kubernetes YAML which is verbose and extremely clear compared to alternatives, but can also be quite ugly and verbose. I think that Kustomize and Kubernetes YAML is not always considered to be the best tool for this type of expressiveness requirement in a configuration language.

The technique that is suggested is to generate Kubernetes YAML into a deploy branch, or into a pull request off of a deploy branch, generated from something more flexible and expressive like Jsonnet; and to do it in CI, ahead of the deploy, actually more like at "release" time. (In my example so far I've used mostly bitnami kubecfg, which is IMHO pretty fairly easy to load into GitHub Actions.)

That way you get the best of both worlds: a whole lot of plain YAML documents, which actually pretty great for reviewing diffs and accepting or understanding changes, knowing what effect it will have to merge a pull request. But also with the use of libraries to get the expressiveness of a programming language.

So you can make custom objects and let them expand to rehydrated form from a small utterance to expansion of a large amount of boilerplate, but not to where a human must rewrite and repeat it ad nauseum over and again.

I think you're right that scaling plain YAML with Kustomize is not always ideal, solving this is a priority for Flux.

[adberger]

In my opinion the Manifest generation of Flux v1 was a really important feature which Flux v2 currently lacks.
We for example would like to use our own tool called kusbile.
It drastically reduces code redundancy over multiple environmnents & clusters.

Thanks for sharing your example on "Manifest generation in a different branch of the git repo".
We currently use GitLab so we would need to write our own implementation to have the same result.
Personally I don't like the generation in the source repo and would like a generation on the fly (like helm does via templating).

The problem described in this issue seems to be a "problem" for almost every GitOps method. Although ArgoCD for example has a plugin system which can somehow help with this.

We will fiddle around a bit more and see if plain YAML (with some Kustomize) is sufficient for our needs.