Dependency file generation for build tools

[btrepp]

Eschewed features

• [X] This issue is not requesting templating, unstuctured edits, build-time side-effects from args or env vars, or any other eschewed feature.

What would you like to have added?

Kustomize build having the ability to generate makefile (or ninja dyndep file) compatible syntax. For the case of makefiles, this would be an extra flag similar to how C compilers work. For the case of ninja dyndeps (which would run before build) this might need to be a seperate subcommand.

This would probably be 2x features, makefiles are more 'universal' but ninja files are more powerful. Even the makefile style would have immediate benefits to many people.

Why is this needed?

It would be advantageous to only run Kustomize when the output would be different in build systems such as ninja and Make. This would lead to increased speed in larger systems, and by making Kustomize build play nice with build systems, it can become a tool that's used inside a greater whole.

With a source file -> kustomization output -> Kubectl apply, a build system can become quite smart to know when to run Kubectl apply and when not to. This pairs well with the new prune --applyset feature in development for Kubectl.

Can you accomplish the motivating task without this feature, and if so, how?

A rough implementation of the dyndep version is here https://git.sr.ht/~btrepp/environment/tree/main/item/tools/kustomize-dyndep

This is non optimal though, as technically I am 're-doing' the logic for finding files from a kustomization.yaml and I haven't yet implemented all the parts, but it has proven to be various useful. While the dyndep lets me do other things (e.g cached bases was the motivation) even a simple makefile deps style would be very beneficial.

What other solutions have you considered?

• Developing an external tool as above.
• Using wildcards in makefiles (error prone), duplicate information in deps, doesn't walk bases very well.

Anything else we should know?

Is there a formal specification for the kustomization files?. That may help in implementing it in another tool. If not it's probably best done in Kustomize itself, as that would keep the shapes up to date.

Feature ownership

• [ ] I am interested in contributing this feature myself! 🎉

[natasha41575]

I don't think I fully understand the feature request, so you can feel free to correct me if I've misunderstood something or if you want to provide additional examples.

It sounds like you are asking for kustomize build to be able to output something in a different format than what it does today, but it's unclear to me what this means as kustomize build currently outputs manifests that can be provided directly to kubectl apply, which is a very specific and well-defined format.

I'm not sure that I understand what your desired "makefile syntax" would look like. We don't really have control over what kind of yaml input kubectl/kubernetes accepts, and it doesn't really make sense for kustomize to output any other kind of format. Perhaps a specific example of what the output would look like might help?

With my limited understanding, this feature request sounds like it might be better placed as its own tool separate from kustomize, so I will mark this as out of scope unless there is more information to suggest otherwise.

Is there a formal specification for the kustomization files?.

Yes, there is. You can find it here: https://github.com/kubernetes-sigs/kustomize/blob/0add0f95e2d7be992801411bb1851c528b9831dd/api/types/kustomization.go#L34

[btrepp]

Apologies. Sometimes when you are in writing up problems you become so normalised to Makefile rules that I forgot to elaborate on them.

As far as external tools, while possible (and I've written a makefile version here https://git.sr.ht/~btrepp/kustomize-depfiles/tree) it's not the best user experience, that this is some other tool.

To write it up more

In a makefile

If I wanted to integrate kustomize into a makefile, I might approach it like so

out/kustomized.yaml: src/kustomize.yaml
     kustomize build src/ -o out/kustomized.yaml

However, that says the output only depends on the kustomization.yaml. In make the output only gets rebuilt if the src changes, but we can't express the source easily. Unfortunately the only real thing that knows what kustomization.yaml depends on (patches/bases/resources etc) is kustomize itself.

This is actually not a unique problem, C code has the same issue, you don't know what headers a C file depends on, unless you parse the file.

From gcc docs https://man7.org/linux/man-pages/man1/gcc.1.html

 -MF file
           When used with -M or -MM, specifies a file to write the
           dependencies to.  If no -MF switch is given the preprocessor
           sends the rules to the same place it would send preprocessed
           output.

So this produces an 'extra' file, for kustomize the expectation that an extra file would be produced too

E.g assuming something

-include $(out/kustomized.d)
out/kustomized.yaml: src/kustomize.yaml
     kustomize build -MF out/kustomized.d src/ -o out/kustomized.yaml

Setup this way, make would run kustomize if no dep files exist, but then once ran the first time, be able to understand what file changes should cause a kustomize rebuild.

This drastically would improve cycle time in using kustomize. In my POC tools it's become significantly faster to iterate over multiple bases as only changes trigger the kustomization to rebuild.

For the output format, here is a sample from my simple tool

out/kustomize.yaml : \
	dist/nginx/v1.25.0/base/deployment.yaml \
	dist/nginx/v1.25.0/base/kustomization.yaml \
	dist/nginx/v1.25.0/base/service.yaml

Which is standard makefile format.

My usage of it is similar to something like

-include $(out/kustomized.d)
out/kustomized.yaml: src/kustomize.yaml
     kustomize-depfiles --target out/kustomized.yaml src
     kustomize build -o out/kustomized.yaml

Which works, but there is a risk I don't implement the file-loader correctly, which does come to the next point.

I appreciate the link the the source types, but that's probably not a specification of the file-format, I imagine this isn't something that is produced yet, as kustomize probably doesn't expect other tools to interact with it so much.

The main motivation for the feature request, is like gcc, to have this internalised so that the resolution and file identification logic stays the same. Implementing it in another tool has highlighted that there is a risk the understanding of the file would diverge. Fine for my purposes, not as great for any users who aren't as familiar with kustomize.

An extra point to make, there are many people asking for 'caching' of remote bases and other cache features for kustomize. Using this with git-submodules for remotes has basically eliminated that need for me, so kustomize probably doesn't need fancy caching capabilities, just the ability to let existing tools understand what files it depends on. So this plus some documentation could satisfy the 'caching' need, without some of the complexity that would introduce.

Also this is only really a request. If kustomize isn't interested in it, that's fine :). I've got a solution that works for me, but wanted to help improve the tool for everyone.

[komapa]

@btrepp How would you handle remote resources, like from a remote git repo?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[colindean]

/remove-lifecycle stale

[btrepp]

@btrepp How would you handle remote resources, like from a remote git repo?

Apologies I missed this reply :).

It's a very good question, to be honest I never had the need, as remote, lazy sources seemed like a security risk to me :), and not very repeatable. However I do have an answer

In my current proof of concept, I ignore remote files, only focusing on local files. Remote files would be ambiguous. Make itself works best when it knows the time stamps of everything.

For a principle of least surprise, we could have the deps file generate .PHONY make targets for remote urls. This might be an issue for ninja, but would be valid in make. This would signify 'always' rebuild, much like currently you would need to do for remote. There is arguable utility to this, having it build all the time isn't really what you want for deps files.

What I do for remotes though, is setup make targets to actually get those files, such that they are cloned (or curl'd, or whatever your file retrieval of choice is). Then make/kustomize get smart enough to know if you 'updated that' (git pull) or something else. The other option (that works great) is git submodules. You just say your base is in this other folder. Updating the submodule will then trigger it to be rebuilt (if the submodule changed). This also works as quite often you do need to fiddle with remotes anyway, most people don't structure them to well to allow kustomize to build on them, so they need to be slightly wrangled to play nice with kustomize, so you can write and patch steps to have it play nice with kustomize.

I will also add on, even though I do have a PoC, I find myself missing this feature heaps when using k8s/kustomize. Even starting something new, I start of with hacky wildcards, but don't handle recursion as well as the PoC did. However with the PoC as I don't have a good formal specification of the kustomize file format, it still has a risk.

I'd be still happy to maintain it in a different repo and tool, but the lack of spec means I have to reverse engineer it, so while do-able, I'd worry about missing cases and having edge cases.

Even if kustomize didn't support a makefile/deps file format, a subcommand that would spew out 'all files associated with this build', could then be adapted to different languages. It might be possible to used strace the build process to accomplish that too. So I might explore that a a shell script.

EDIT: Ah I now remember the deficiency with as strace approach. It will only work if all the files already exist. This would be normal... most of the time, but I wanted to do things like generate secrets using shell commands, or retrieve them from a secret management system (eg a makefile recipe for the 'secret file') and avoid checking them into source control.

Strace won't really be aware of the files that don't exist, so if you need to do something outside of kustomize to generate them, it will be difficult

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[btrepp]

Note: I did have a stab at attempting this myself, however the internal code of kustomize was a bit difficult for me to navigate, I suspect also the effect of 'multiple layers of loading files lazily', makes it not straight forward for a beginner to the code-base to add.

There is still value in this, despite the stale bot, however would need pointers if the community is to implement it.

[btrepp]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten