Kubectl Kustomize support

[tobbbles]

Since Kubectl now supports native kustomize building through the following flag -k, --kustomize='': Process a kustomization directory. This flag can't be used together with -f or -R., it'd be really nice if I were able to specify and utilise this in my k8s_object build step.

I'm unsure what the API would look like, as -k isn't available in kubectl options.

[nlopezgi]

Hi @tobbbles , thanks for opening this issue. rules_k8s relies mainly on community contributions for new features. I'd be happy to work with you if you want to propose a design to implement support for your desired feature. Thanks again,

[TannerYoung]

Found this and I think it would be a huge benefit to the rules_k8s ecosystem.

I think the difficulty implementing this for someone not familiar with how the original was written will be that it is currently set up to handle single objects (but really a full source file) at at time, and not directories. With the kustomize kubectl command you pass a specific directory which contains a kusotmization.yaml, and that file may reference other files in the same directory or the kustomization.yaml in a base directory (which also references files in the base directory).

From the bazel perspective, the rule would at the least need to depend on the kustomization.yaml, and a rule from the base. From a user perspective it could look like:

k8s_kustomize(
  name = "base_kustomization",
  srcs = [
     "kustomization.yaml",
     "deployment.yaml",
     "config-map.yaml",
     ...
  ]
)
// this in another directory
k8s_kustomize(
  name = "staging_kustomization",
  deps = [
    "//base:base_kustomization",
  ],
  srcs = [
     "kustomization.yaml",
     "deployment.yaml",
     "config-map.yaml",
     ...
  ]
)

But potentially instead of relying on files it would want to rely on other k8s_object? I'm not 100% sure what some of the intentions of the original design were, and how this approach might be breaking them.

Definitely going to need some guidance from the original author for anyone to be able to move forward with an implementation proposal.

[erain]

I have mixed feelings about having kustomize with rules_k8s.

• kustomize is a useful tool in k8s ecosystem..
• However, the current implementation of rules_k8s is pretty much container image centric with template yaml file. kustomize, on the other hand, relies a lot on the yaml files patching and itself is quite complicated.

I would advice to design and prototype a new rule rules_kustomize for just producing yamls, and the existing rules_k8s for applying the yaml.

[tobbbles]

I feel a new rule that's responsible for the dependency tracking and resolution of bases, resources, and patches, with kustomization.yaml being the entry point would provie useful, with the perceived 'output' of this rule being a built YAML bundle (similar to what kustomize build gives us.)

One concern I have is with kustomize's remote targets, and how these would be handled and behave in the rule.

[tobbbles]

would advice to design and prototype a new rule rules_kustomize

Is there an outlined process for going through the design and prototyping of such a rule?

[erain]

I don't have an "official" answer for the process, but I would start simple and create something that is working, and see how people like it. To be honest I think that was how rules_docker started.

My only advice for the rules_kustomize is that its output can be directly placed into the template field of k8s_deploy.

[rverma-jm]

Hi folks, do we have any workable solution for this?

[joeljeske]

Rules_gitops has an approach for using kustomize that seems interesting. I do agree though that kustomize should not be part of this project, but a different set of rules.

[alexeagle]

kustomize is part of kubectl so IMO it's reasonable to add it to rules_k8s. More rulesets means more maintenance burden, questions about governance, and more dependencies for users to juggle.

[tonybenchsci]

Hi folks, today we're open sourcing https://github.com/benchsci/rules_kustomize We've been using a version of this with native k8s and k8s-Config-Connector CRDs in a monorepo.

[dymart]

Hello! I'm trying to create a set of rules for kustomize ( https://github.com/ubiquitoustech/rules_kustomize ). They currently have basic functionality but I would like to allow for generated files to be consumed as deps by the kustomize rules as well. The motivation for this is so that tools upstream of these rules can generate configs and kustomize can still consume them.

If anyone has any ideas or knows of a way to move or symlink generated and source files together so they can be consumed as kustomize expects inside of ctx.actions.run that would be appreciated. Or if there is a better approach please let me know. I'm happy to try and implement ideas or known solutions to this if anyone has any.

https://github.com/ubiquitoustech/rules_kustomize/blob/main/examples/deps/BUILD.bazel is a simple example of what I hope the rule could function like.

Hopefully a set of rules like this would be complimentary to rules_k8s and help resolve this issue. thanks