kustomize
kustomize copied to clipboard
kubernetes-sigs
Enable globbing in "resources" field of kustomization.yaml
This is basically the same as https://github.com/kubernetes-sigs/kustomize/issues/119. Reopening, as the issue was closed. The reason for reopening is that we (Arrikto) believe that the reasons given for eschewing this feature don't really apply.
It's also interesting that other use-cases mandating glob-like use have come up from the Kustomize team (although I feel the particular one could benefit from more explicit globbing): https://github.com/kubernetes-sigs/kustomize/issues/3204
Is your feature request related to a problem? Please describe.
I want to automatically pick up new kustomizations inside a folder. For example, we have a structure like this:
user-resources
├── user1
└── user2
...
and I want to have a kustomization file that automatically imports everything under user-resources:
resources:
- path/to/user-resources/*
Describe the solution you'd like
Kustomize should support globbing in resources. This feature is eschewed right now and the given reason is:
kustomize supports the best practice of storing one’s entire configuration in a version control system.
Globbing the local file system for files not explicitly declared in the kustomization file at kustomize build time would violate that goal.
Allowing globbing in a kustomization file would also introduce the same problems as allowing globbing in java import declarations or BUILD/Makefile dependency rules.
kustomize will instead provide kustomization file editting commands that accept globbed arguments, expand them at edit time relative to the local file system, and store the resulting explicit names into the kustomization file.
I'd like to elaborate why we are not very clear on those arguments.
kustomize supports the best practice of storing one’s entire configuration in a version control system.
Globbing the local file system for files not explicitly declared in the kustomization file at kustomize build time would violate that goal.
This is not clear to us. Storing the entire configuration in a VCS means that checking out a specific commit and running kustomize build should produce the same result everytime. That's why kustomize has explicitly avoided CLI arguments and environment variables. But in the case of globs, the result of globbing the filesystem of a specific commit will always be the same. There is no hidden state here.
Allowing globbing in a kustomization file would also introduce the same problems as allowing globbing in java import declarations or BUILD/Makefile dependency rules.
The problem mentioned in the referenced blog post is that when importing with globs, name collisions may occur. The example given in the blog post is that if one writes:
import java.util.*;
import java.awt.*;
// Where did the List class come from?
List choices = new List();
and the List class is declared by two imports, Java will silently keep the one declared later in the code.
However, this doesn't really apply to kustomize, because:
- Kustomize forbids resource collision (same group, version, kind, namespace, name).
- Kustomize imports are not very useful for deterring collisions. If one follows the suggested best practice and imports both
java.util.Listandjava.awt.List, then it's immediately clear there is a collision, so catastrophe avoided. However, if one importsa/deployment.yamlandb/deployment.yaml, it's not really clear if there is a resource collision.
Thus, we believe this drawback doesn't really exist for a tool like kustomize.
Describe alternatives you've considered
Kustomize provides the alternative of CLI commands that support globbing and can refresh the kustomizations. However, this imposes an additional burden on the user, to track every single place where this must occur.
I'd love to hear your thoughts on this @monopole @Liujingfang1 @Shell32-Natsu Also cc'ing the author of the original issue @ahmetb
It's become clear over time that kustomize is primarily used in a git context.
When that's the case, questions about what went into a build are moot, and a resources: field can be globbed or even eliminated as suggested in #3204.
Also update docs site to describe how to use globs to manage the goal here.
I have to say I am surprised by this issue. I just started learning to use kustomize yesterday and it seemed like it would perfectly fit in my workflow. I have a directory of all my YAML files in git and I simply kubectl apply -f myyaml/ on the directory to recursively apply all the YAML there, and kustomize in kubectl seems exactly like a drop-in alternative kubectl apply -k myyaml/ but with the benefit of consolidating and simplifying my YAML management.
It seemed so intuitive and obvious that
bases:
- ../baseyaml
should apply all the YAML in that directory, just like kubectl apply -f , that I was very confused about why you would need to explicitly specify any .yml files as resources in kustomization.yml in the base directory since this would be redundant. This is in all the examples but the documentation does not explain why. I thought you needed to specify a resource in the base layer only if you want to allow a higher layer to overwrite those values, and that otherwise it would be "read only" (but obviously pulled in by default to the upper layers) because I could not think of any other way this would make sense. It was so counterintuitive it took awhile to figure out how mislead I was.
So, now I download yet another tool (kustomize) , dig up an ancient version to avoid https://github.com/kubernetes-sigs/kustomize/issues/1342 so it doesn't break kubectl apply -k , run another command (kustomize edit add resource *.yml) , clean up after it because *.yml matches kustomization.yml and it didn't know not to add itself as a resource in kustomization.yml , and then I end up with uncommitted changes made to kustomization.yml in my version control as a result of this extraneous process (so it does not really seem that conducive to a git-based workflow IMHO), when all I wanted was to do what is (from my point of view) obviously implied by specifying a directory as a base in the first place: just use the YAML in that directory. In hindsight I should have just scripted up something dumb like this: https://github.com/kubernetes-sigs/kustomize/issues/119#issuecomment-526234246
kustomize supports the best practice of storing one’s entire configuration in a version control system.
Globbing the local file system for files not explicitly declared in the kustomization file at kustomize build time would violate that goal.
From my point of view I want a way to explicitly declare a directory to use as a base. And when I manage the contents of that directory in version control, it is not at all clear how loading all the YAML there would violate the best practice of storing one’s entire configuration in a version control system (in fact precisely the opposite, it seems to closely conform to that goal). It is my responsibility to make sure I have the right YAML contents in my version control , not kustomize's responsibility. kustomize should not have opinions about how I specify or manage my files; it should do what I say , like any tool. If I accidentally break my deployment I will blame myself but if kustomize doesn't let me easily manage and apply YAML maybe I should blame kustomize :) Well I won't blame it , I made some incorrect assumptions and my intuition and expectations may have been misguided, but I hope it may be productive to describe my initial (maybe naive) user experience and perspective running into this issue. Thank you!
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale
Being unable to use wildcard in "resources" field in kustomization.yaml adds an extra burden when trying to implement GitOps principles.
Imagine you want your users to be able to simply add a manifest to the repository and this way the manifest gets automatically deployed in the environment. That's great! And simple.
Now imagine this procedure has changed and you must tell this same user to edit some file called "kustomization.yaml" (that he doesn't know what it is) and to add a new entry pointing to this new manifest he has just added. Or better, you tell him to download the kustomize CLI (he still doesn't know what it is) to do this step in a "simpler" manner. Or you integrate it to your pipeline by adding a hook to automatically make a new commit updating just the "kustomization.yaml" with the new entries.
In my view it adds extra unnecessary complexity and introduces new "points of failure" to the whole gitops flow.
I have a usecase where I control everything using flux and kustomize files, but deep down in the structure (cluster/namespaces/core/....) is a folder which is a git submodule without a kustomization.yaml file. I want that folder there for structural purpose, however, I can't include it without needing to look at the files in the parents kustomization.yaml file from time to time.
It would be better if I could glob-import them
@KnVerey @pwittrock Don't you guys think this basic obvious requirement, should be prioritised?
What the hell is wrong with Kustomize maintainers. Lacking this basic feature is ridiculous, Open this issue up.
@arash-bizcover I understand the frustration but we are 3 people with jobs and competing priorities outside of kustomize - we are addressing issues as quickly as we can. You are welcome to submit a PR or bring this issue up at a SIG-CLI meeting if you want more traction on this.
The idea is similar to having a /etc/something/conf.d/ directory, which is a widely adopted design pattern where you can just drop any file in in order to include a modular piece of configuration.
Expressing interest in the feature - in a respectful and supportive way - may be helpful, but perhaps it could also help to have an indication whether the maintainers would accept/approve this feature. If so perhaps someone could be forthcoming with a PR.
Personally I am open to the idea as there seems to be a lot of user interest and I can see use cases that prevent kustomize edit add resource /etc/something/conf.d/* followed by kustomize build to be a bit of a hassle - especially the cases described by @xeor and @hvitoi above.
However, I would like some more input from @KnVerey and @monopole because such a feature is specifically defined as out of scope for kustomize (https://kubectl.docs.kubernetes.io/faq/kustomize/eschewedfeatures/) and many of the decisions we have made recently have been to avoid these eschewed features - and I'd like some more context behind the decision to firmly disallow globbing.
Notably listed in this issue: https://github.com/kubernetes-sigs/kustomize/issues/3204:
Implicit inclusion violates the benefits of an unambiguous manifest declaration (see this note on globbing). This is mitigated by always working in the context of a git repository (so you can always know how or if what you have deviates from the repository contents).
So another option may be to enable globbing only when working in a git repository.
@KnVerey @monopole Do these use cases seem compelling enough to make an exception, perhaps as a configurable option?
yes this usecases are compelling enough to make an exception in my opinion :) this feature would be great because in a cd/cd pipeline view you want to add a new resource automated, to edit the kustomization.yaml too with every new file is more complicated then just pushing a new file in my opinion
After careful consideration of the input on these issues, I'm inclined to agree that limited glob support in the resources field could provide concrete benefits and carry reasonably minimal risk in the use cases we should be optimizing for, i.e. settings in which GitOps best practices are being followed.
That said, I do think that Kustomize should continue to favour being extremely explicit in general. If we implement globbing, we should tightly scope this feature and update the eschewed features document accordingly. For instance, I personally think #3204 should still be out of bounds as inadequately explicit to be a fit for Kustomize, and I would prefer a smaller, less flexible feature over a more powerful but less predictable one.
We have a brand new process for aligning on features that are large and/or particularly contentious before time is spent on implementation: a mini in-repo KEP. If someone following this issue still feels strongly about it and is interested in contributing an implementation, please create an in-repo KEP with a more detailed proposal around what globbing will look like, how it will work, and what the boundaries of the functionality will be.
For example, please include:
- When a given directory is targeted with a glob, what exactly does Kustomize do?
- bare minimum is accumulate the resources from all .yaml files in the immediate directory
- probably .yml as well
- probably not .json or dotfiles that end in .yaml
- definitely not automatically recursive
- what if the directory contains other files? What if the glob expression attempts to explicitly target them (e.g. resources/*.yaml.foo)?
- what if the directory contains a Kustomization?
- bare minimum is accumulate the resources from all .yaml files in the immediate directory
- Will deeper directory structure targeting be supported?
- Will pattern matching be supported? To what extent and why?
- Will it only work in the
resources:field? What abouttransformers:for example? - Will this feature be gated (i.e. flag- or field-enabled), either during alpha or permanently?
Per my strong preference for a tightly scoped feature, my default stance on most of the above is "should not be supported", but I'm open to having my mind changed by concrete use cases in the KEP.
- When a given directory is targeted with a glob, what exactly does Kustomize do?
- bare minimum is accumulate the resources from all .yaml files in the immediate directory
bare minimum should be:
which results in all .yaml files in this naming pattern picked up without changing the kustomization.yamlresources: - api/spring-boot-api-*-custom-resource.yaml - api/go-gin-gonic-api-*-custom-resource.yaml- probably .yml as well A: set defaults, let user override it, default .yml/.yaml, user can override to .yml.foo/.yaml.foo if wanted/needed
- probably not .json or dotfiles that end in .yaml A: solved via default yml/yaml and user overrides if needed or wanted, but excluding dotfiles would be good i think
- definitely not automatically recursive
why not, someone could want to do, if he operates 10 projects, with 56 sub-projects, with 80 spring boot api's and 46 go api's located in those subprojects:
resources: - api/*/*/spring-boot-api-*-custom-resource.yaml - api/*/*/go-gin-gonic-api-*-custom-resource.yaml
- what if the directory contains other files? What if the glob expression attempts to explicitly target them (e.g. resources/*.yaml.foo)? A: solved by above, default .yaml/.yml if user overrides to yaml.foo it should be possible to explicitly target them like this
- what if the directory contains a Kustomization? A: don't now the impact of this but would it be possible to just include it when the naming pattern matches?
- bare minimum is accumulate the resources from all .yaml files in the immediate directory
bare minimum should be:
- Will deeper directory structure targeting be supported? A: would be a good idea i think
- Will pattern matching be supported? To what extent and why? A: i think it is needed, why, because of the above
- Will it only work in the resources: field? What about transformers: for example? A: in a first step it would be enough to support this in resources i think
- Will this feature be gated (i.e. flag- or field-enabled), either during alpha or permanently? A: i dont think it needs to be gated to be honest
/remove-kind design
/kind feature
kind/design is migrated to kind/feature, see https://github.com/kubernetes/community/issues/6144 for more details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
Please accept wild cards on resources. If we have 200 resources on a folder that we want to load we need to specify it on the Kustomize.yaml which duplicates file names all over.
One reason that it is useful is to avoid git conflicts on kustomization.yaml. Conflict handling is usually a manual process. CI/CD pipelines that modify the kustomization.yaml file will not be able to do that. kustomize edit add resources ... will not handle this use-case.
I came here looking for some globbing support. I understand that globbing can have it's own pitfalls. I think it's a bit counterintuitive to not even support resources as a folder. I mean if bases support specifying a folder like ../../base, then why can't we simply have resources: ['somefolder'] so we don't have to explicitly call out every single YAML file? If there are other unsupported files, then ignore them and leave it up to the user to clean that up. But it seems like a logical request, no?
While I'd love a native solution to this, instead I created a little Python wrapper around kustomization.yaml that will walk sub-directories and populate every desired file into a kustomization.yaml, as well as ignore any number of files (if you wish). It's totally 'solved' this problem for me for over a year so I don't have to manually manage and diff a ton of file paths. Ain't nobody got time for that.
https://github.com/DaemonDude23/helmizer
Better to say 'People still want globbing!'
Your best practices are not everyone's best practices.
What the hell is wrong with Kustomize maintainers. Lacking this basic feature is ridiculous, Open this issue up.
@arash-bizcover I understand the frustration but we are 3 people with jobs and competing priorities outside of kustomize - we are addressing issues as quickly as we can. You are welcome to submit a PR or bring this issue up at a SIG-CLI meeting if you want more traction on this.
I get that the tone wasn't very kind. But wasn't this already implemented once?
Why was this even removed?
Why does someone's opinion mean others have to put considerable amount of effort into figuring out an alternative to putting a *.
Might as well remove globbing from bash...
For a lot of automation it is much simpler and faster to just drop a file in place instead of having to drop the file in place and update the kustomization.yaml, which has much greater potential to go wrong.
What is the current stance on this? Is there opposition to adding globbing, or is the problem that no one has time to add it?
