flux2

flux2 copied to clipboard

Feature Request

Flux Kustomization should allow .path pointing to individual files and not just folders

The .path field in flux's Kustomization CRD must currently be a folder.

This seems to be because flux internally looks for a kustomization.yaml file in the folder, and if it doesn't exist it creates a dummy one to support non-kustomization based yaml.

We think this behaviour can be preserved even if .path points to an individual file by putting the file into some temp directory and creating the dummy kustomization.yaml in there.

Why we want this feature

1. Allowing only folders as .path is a confusing requirement, and it leaks the internal interface (of using kustomize build internally) to the user, and we feel this is confusing if kustomize-controller is meant to handle arbitrary raw YAML, but then requires a particular nested folder structure.

2. It clashes with our existing gitops setup for fluxv1 (one large file per app) as propagated through packaging tools (expanded on below). We think that this is a reasonable setup, and migrating our packaging solutions to move files across repo boundaries would be a painful requirement to roll out "just to upgrade flux".

How we think this feature could be done

If the .path exists as a file, we could follow an alternate way of creating the dummy kustomization.yaml by moving the referenced file into a designated/temporary directory first.

Existing application and kubernetes manifest setup

Multi-tenant setup with each namespace being managed by a repository of yaml, and many teams committing to that repository in an automated fashion via a packaging tool.

The packaging tool squashes any application yaml (rendered on the application side via kustomize/helm) into a single file (with many resources) for that application. In the end, the gitops repository ends up looking like this:

gitops-repo
├── development
│   ├── application1-manifest.yaml
│   └── application2-manifest.yaml
└── production
    ├── application1-manifest.yaml
    └── application2-manifest.yaml

Sample gitops repo for reference

cc: @tl-eirik-albrigtsen @felipesere @lukemathwalker @tl-marco-ieni

@suryapandian can't you change your manifest generator to create a dir with the app name?

From:

• development/application1-manifest.yaml

To:

• development/application1/manifest.yaml

As a work around we could do that. However, it is disruptive to the existing workflow. We would have to change the manifest generator, but we would also have to deal with many issues:

• garbage collecting old manifests
• forcing many old applications to redeploy and potentially upgrading their CI to be able to do so.

We could also use gitrepo crd instance by excluding filepaths as Kingdon suggested. This would make the flux configuration spammy by creating one gitrpeo crd instance and one kustomization crd instance for each application manifest.

Would a non-kustomize manifest apply controller be of use for this use case, particularly if it supported a specific list of files to apply?

I'm running into a somewhat similar issue where I have a set of manifests that I want to apply to a cluster, and the most logical way of organizing those files is a positive selector list rather than an exclude pattern. I suppose I could do:

# Start by excluding everything
*
!**/my-cluster/local-config.yaml
!**/base/app.yaml

Where my-cluster might vary from one cluster to another, and local-config.yaml contains per-cluster config (like Ingress with hostnames) while base/app.yaml is a common set of core features.

@kingdonb , since he was in the Slack thread. I decided to follow up here so the record would be easier to find.

I think the behavior we are pushing for users to adopt is to make more complete artifacts, not more fragmented ones; since there is a very tenuous relationship between the hypothetical Kustomization that points at several files in specific locations and those files and their locations, there is a whole new class of errors we will have to contend with when one of the requested assets is unavailable; should the remaining resources be applied if they are valid without it?

It's much better to have one asset and apply it, or apply from a specific sub-path, and have those applies be atomic. Then the asset should also be formed in an "atomic" way where you cannot lose a part of it.

When OCI support is released in Flux 0.32.x, the recommended course of action for people who want to keep many resources in the same directory and have them all synced separately would probably be to set up a CI job that monitors for changes to the files in that directory, and runs flux push to generate a new artifact for Flux to apply when they change.

You can still use one OCI artifact just as you used one gitrepo, but the internal structure will need to be expanded so that each YAML resource gets its own folder path. This can be done in a procedural way by the CI job so nobody ever needs to look at a tree of resources, you just replace app-env-my-instance-1.yaml in your head with app-env-my-instance-1/ for example, the file is moved to a directory that matches its base name... and the CI job that pushes them should take about 15s under ideal conditions. It can be very simple in terms of logic.

We are very careful about making changes that add or remove functionality to the design of Flux because of the stability requirements of production-grade infrastructure. If we enable some behavior that was not possible before, we need to consider all the possible consequences. You can almost make this change in your CI today without any committee involvement. (The OCI feature has been in the works for a long time, it went through this RFC process and has been developed to the spec 👍 it should solve a lot of problems like this and may even fully replace Image Update Automation in the long run.)

A feature request is the right way to start to go about this, but if it is deemed a substantial change, it will need to go into an RFC, and then be approved by the Flux maintainers. This is to give the community enough time to respond to design changes that aren't completely well-thought-out and get them into a state that we can consider acceptable as a permanent condition for the future of Flux and maintainable for ten years or more.

I think what both @suryapandian and myself are looking for might be a KubeApply resource and controller. I agree that adding a new top-level CR adjacent to Kustomization and HelmRelease is probably worth an RFC if that's actually the feature request. 😁

Personally, I don't have a good sense of how many other people want that; I'm just here to add my voice.

My particular scenario is that I'm looking to extend some existing tooling for cross-cluster environments that's not necessarily bought into either kustomize or helm. A typical setup may look like (not exact, a synthesis of several different solutions):

$team/cluster-a/clusterconfig.yaml  # Contains HPA, Ingress, ConfigMap for cluster A
$team/cluster-b/clusterconfig.yaml  # Contains HPA, Ingress, ConfigMap for cluster B
$team/staging/clusterconfig.yaml    # Contains HPA, Ingress, ConfigMap for staging
$team/bases/resources.yaml          # Base app definitions: Deployment, operator resources, etc.

The goal would be that a given cluster applies one $team/<cluster> set of configs and the contents of $team/bases; we would be okay with a GitRepo and an "apply" resource per team+cluster combo, but would like to minimize the amount of additional "glue" configurations in the Git repo. It would be possible to introduce kustomization.yaml files for each $team/<cluster> directory into this configuration, but adding two new tools (Flux and Kustomize) feels more difficult than adding one.

The goal would be that a given cluster applies one $team/ set of configs and the contents of $team/bases

So in that scenario your teams would always have to create two KubeApply resources, one for <cluster> and another one for bases whereas with kustomize you'd have one with the added benefit of it being kind of an atomic operation.

As much as I understand that with Kustomize your teams would have to learn "just another tool", it can be used in a very very simplistic way.

The idea of introducing a KubeApply resource doesn't resonate very much with me as it would make Flux more complex with very little benefit. But that's just my personal opinion and you're very welcome to create an RFC PR elaborating on your idea a little more.