Kustomization PreBuild Stage

[oliverbaehler]

I have a infrastructure use-case where I want to consider different factors (location, segment and environment) to deploy a cluster. Each of these factors can influence for example values with patches or whatever. For now I have implemented something like this:

---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: cluster-bootstrap
  namespace: flux-system
spec:
  interval: ${kustomization_interval:-${global_interval:-5m}}
  path: ./infrastructure/location/${location}/segment/${segment}/environment/${environment}
  prune: false
.....

The only problem being, it's not super scaleable. Since I have to consider each possibility of the named factors I have to create for each outcome a dedicated folder. Which is quiet messy and if i add something new like a new environment I have have to create hundreds of folders. to be concrete, the structure would look something like this:

infra/
├── environment
│   ├── dev
│   │   ├── kustomization.yaml
│   │   ├── operators
│   │   │   └── sealed-secrets
│   │   │       └── release.yaml
│   │   └── test.yaml
│   ├── prod
│   │   └── kustomization.yaml
│   ├── stage
│   │   └── kustomization.yaml
│   └── test
│       └── kustomization.yaml
├── location
│   ├── a
│   │   ├── ci
│   │   │   └── environments
│   │   │       ├── base
│   │   │       │   └── kustomization.yaml
│   │   │       ├── dev
│   │   │       │   └── kustomization.yaml
│   │   │       ├── prod
│   │   │       │   └── kustomization.yaml
│   │   │       ├── stage
│   │   │       │   └── kustomization.yaml
│   │   │       └── test
│   │   │           └── kustomization.yaml
│   │   ├── co
│   │   │   └── environments
│   │   │       ├── base
│   │   │       │   └── kustomization.yaml
│   │   │       ├── dev
│   │   │       │   └── kustomization.yaml
│   │   │       ├── prod
│   │   │       │   └── kustomization.yaml
│   │   │       ├── stage
│   │   │       │   └── kustomization.yaml
│   │   │       └── test
│   │   │           └── kustomization.yaml
│   │   └── kb
│   │       └── environments
│   │           ├── base
│   │           │   └── kustomization.yaml
│   │           ├── dev
│   │           │   └── kustomization.yaml
│   │           ├── prod
│   │           │   └── kustomization.yaml
│   │           ├── stage
│   │           │   └── kustomization.yaml
│   │           └── test
│   │               └── kustomization.yaml
│   └── b
│       ├── ci
│       │   └── environments
│       │       ├── base
│       │       │   └── kustomization.yaml
│       │       ├── dev
│       │       │   └── kustomization.yaml
│       │       ├── prod
│       │       │   └── kustomization.yaml
│       │       ├── stage
│       │       │   └── kustomization.yaml
│       │       └── test
│       │           └── kustomization.yaml
│       ├── co
│       │   └── environments
│       │       ├── base
│       │       │   └── kustomization.yaml
│       │       ├── dev
│       │       │   └── kustomization.yaml
│       │       ├── prod
│       │       │   └── kustomization.yaml
│       │       ├── stage
│       │       │   └── kustomization.yaml
│       │       └── test
│       │           └── kustomization.yaml
│       └── kb
│           └── environments
│               ├── base
│               │   └── kustomization.yaml
│               ├── dev
│               │   └── kustomization.yaml
│               ├── prod
│               │   └── kustomization.yaml
│               ├── stage
│               │   └── kustomization.yaml
│               └── test
│                   └── kustomization.yaml
└── segment
    ├── ci
    │   └── environments
    │       ├── base
    │       │   └── kustomization.yaml
    │       ├── dev
    │       │   └── kustomization.yaml
    │       ├── prod
    │       │   └── kustomization.yaml
    │       ├── stage
    │       │   └── kustomization.yaml
    │       └── test
    │           └── kustomization.yaml
    ├── co
    │   └── environments
    │       ├── base
    │       │   └── kustomization.yaml
    │       ├── dev
    │       │   └── kustomization.yaml
    │       ├── prod
    │       │   └── kustomization.yaml
    │       ├── stage
    │       │   └── kustomization.yaml
    │       └── test
    │           └── kustomization.yaml
    └── kb
        └── environments
            ├── base
            │   └── kustomization.yaml
            ├── dev
            │   └── kustomization.yaml
            ├── prod
            │   └── kustomization.yaml
            ├── stage
            │   └── kustomization.yaml
            └── test
                └── kustomization.yaml

How would you achieve something like that? It kinda doesn't really feel right in my opinion. I thought it might make things easier if I could use variable substitution within kustomize.config.k8s.io files. Then I could do something like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base/bootstrap/sync.yaml
patchesStrategicMerge:
- ../../../../environments/${environment}/
- ../../../../segment/${segment}/
- ../../../../location/${location}/

Currently there is only the option to substitude after the kustomize is build. I would love to add the feature, that there is the option to enable variable substitution preBuild. So the above would work. I am not sure if it's even possible. But I would love to give it a try. But what do you think about it? Maybe you have different opinions on this issue.

I would be more than happy to do the contribution. But only if you agree that is something you would like to see.

[vcariven]

I would be more than interested by this feature too. Starting dealing with more than a few different environments when provisioning infrastructure rapidly leads to a lot of duplicate declarative code, with the only variation of a few strings representing the environment.

Without asking a full templating solution, being able to benefit from the substitution mechanism at the kustomize.config.k8s.io/Kustomisation level would allow having a much more elegant repository structure with less duplicate code

Adapting the existing postBuild syntax to a new preBuild one as suggested by @oliverbaehler and applying the existing substitution algorithm to kustomisations before the build step would add significant flexibility.

Example (Synchronisation)

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
...
preBuild:
  substituteFrom:
    - kind: ConfigMap
      name: cluster-vars

Example (Kustomization)

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ./base/namespace.yaml
  - ./base/helmrelease.yaml
patchesStrategicMerge:
  - envs/${env}/patch.yaml

nameSuffix: ""

generatorOptions:
  disableNameSuffixHash: true

secretGenerator:
- name: my-certs
  namespace: my-namespace
  files:
  - tls.crt=envs/${env}/certs/tls.crt
  - tls.key=envs/${env}/certs/tls.key
  type: "kubernetes.io/tls"

@stefanprodan Would you be open to contribution in this direction ? Or would it break the spirit of the whole thing (flux + kustomize), in your opinion ?

Regards

[stefanprodan]

The issue with preBuild is that we don't know what files should be included, this data is only available after the controller runs kustomize build and gets a stream of all manifests. If we only replace the vars in spec.path then manifests coming from elsewhere, like base dirs and remote https locations will be skipped. Given that Git repos can have GB of data and millions of files, we can't just load all of that into memory and replace vars in any file...

[vcariven]

I think there has been a misunderstanding. My point is simpler than you think. I will try to be clearer.

preBuild would apply only on the kustomisation.yaml file (kustomize.config.k8s.io/Kustomization) if present in the spec.path. It just allows more flexibility at this level. I'm not talking about applying subtitution on all ressources referenced by the kustomisation.yaml (local or remote, before the the kustomize build step). I totally agree with your arguments against that. Moreover, it does not replace the postBuild phase which applies on the yaml manifest stream resulting of the kustomize build.

Example steps:

1. define a kustomize.toolkit.fluxcd.io/Kustomization with a spec.path
2. if a kustomization.yaml is present in spec.path and preBuild.substitute(From) has been defined, apply substitution on this file only
3. call kustomize build on this transformed kustomization.yaml to generate the yaml manifest stream
4. if postBuild.substitute(From) is defined, apply substitution to the generated the yaml manifest stream

I hope I have been clearer.

[oliverbaehler]

@vcariven Thanks for clarifying. I described it on purpose a bit loose, since maybe someone would have better idea. Obviously. we can't support that for the kustomize build (cascading), since we would have to change the kustomize functionality. But on the entry Kustomization we have the possibility to inject variables and that would already help me. I will try to implement it.

[oliverbaehler]

Here's the structure I have used to test my version on: https://github.com/oliverbaehler/kustomize-test

(Very simplified)