Allow kustomize plugins to be optionally enabled

[polivbr]

It would be very useful to allow the new-ish kustomize plugin functionality to be optionally enabled. I understand that there are potential security issues around plugins and that it's an alpha-level feature, so I wouldn't enable by default, but in a tightly controlled environment, they can be extremely useful.

[stefanprodan]

I don't think this is possible due to the fact that the controller uses kustomize as library and the Go plugins will faild the compilation conditions, more details here https://kubernetes-sigs.github.io/kustomize/guides/plugins/goplugincaveats/

As for exec plugins, those should work if you install them inside the container at build time, this is not possible at runtime, since the controller rootfs is readonly due to security reasons.

[polivbr]

If you allow the plugin home directory to be configurable (which I see can be done through the api), it could be pointed at a directory inside of the source, which would allow plugins to be stored in source control alongside the kustomizations. This would get around the issue of the readonly rootfs.

If I can find some time, I'll try to pull together a proof of concept version.

[stefanprodan]

If you allow the plugin home directory to be configurable (which I see can be done through the api), it could be pointed at a directory inside of the source

Ah yes this would be the best approach 💯

[seaneagan]

Note that kustomize plugins can now be defined as KRM config functions which are container based, and thus don't require the plugin home directory mapping, and should just come for free (assuming plugins are enabled, new enough version of kustomize, etc). It's likely that the other plugin runtimes will be deprecated in favor of KRM config functions, in which case it may be better to avoid adding support for them.

[hiddeco]

@seaneagan kpt is a client side tool, that can perform different magic tricks than controllers living within Kubernetes. See my comment here: https://github.com/fluxcd/helm-controller/issues/104#issuecomment-704392154

[seaneagan]

@hiddeco good point, configuring docker-in-docker (in kubernetes) should be possible, but much trickier than simply running plugins in an earlier build step before Kubernetes is involved. Note that KRM config functions are not just a kpt feature, they are also targeted to kustomize (currently behind a flag).

[hiddeco]

configuring docker-in-docker (in kubernetes) should be possible, but much trickier than simply running plugins in an earlier build step before Kubernetes is involved

"Just because you can, doesn't mean you should."

Note that KRM config functions are not just a kpt feature, they are also targeted to kustomize (currently behind a flag).

Had already noted this, but kustomize is a client side tool as well, so it changes very little to what I said.

It's likely that the other plugin runtimes will be deprecated in favor of KRM config functions, in which case it may be better to avoid adding support for them.

I can not find anything to support this claim (and it looks like the issue you link to refers to deprecating the support for kustomize plugins from a kpt perspective), quite the opposite actually, there are discussions to remove the alpha feature flag for plugins.

[seaneagan]

kustomize is a client side tool as well

sure i just meant that folks using the kustomize-controller may expect to be able to use kustomize features, so if there are features it can't support it would be good to document them

It's likely that the other plugin runtimes will be deprecated in favor of KRM config functions, in which case it may be better to avoid adding support for them at some point.

I can not find anything to support this claim (and it looks like the issue you link to refers to deprecating the support for kustomize plugins from a kpt perspective)

AFAICT the plan was to deprecate kustomize plugins in the next major version, while retaining support for KRM config functions, but could be wrong (and plans could change). I think there is a lot of overlap between folks working on kustomize and kpt so github issues get intermingled.

there are discussions to remove the alpha feature flag for plugins.

I don't see any comment from the maintainers yet on that one

[hiddeco]

It took me quite some time to find the right place to look for answers, but finally found what we are looking for in the sig-cli meeting notes.

[yanniszark] kustomize functions vs transformers/generators.

• What is the way forward? Are transformers/generators deprecated?
• kpt functions vs kustomize functions. Different or the same?
• Declarative way to run functions. How will it work with kustomize?
• Answer:
  • KRM functions are the way forward for extending kustomize
  • Function types: go-plugins (deprecating), exec plugins (generators + transformers), image functions
  • Need documentation on how to use each and tradeoffs between 2

From this (and the verbal explanation given during the meeting), I conclude the following:

• Go plugins will be deprecated in the future
• Kustomize will continue to support "exec plugins" (the thing referred to in this issue) and "image plugins" (KRM config functions)

[scottrigby]

sure i just meant that folks using the kustomize-controller may expect to be able to use kustomize features, so if there are features it can't support it would be good to document them

For our docs improvement initiative I was thinking about a similar thing (for both kustomize-controller and helm-controller). specifically, foregrounding in flux docs that the controller supports a subset of features from the underlying tool, and more may be added according to the community needs. We could clarify in a table or something which of these are currently supported. I do not think we should document all missing features though, unless we specifically decide NOT to support one for a specific reason (such as this example, due to technical reasons). Otherwise our docs will always be behind, as new features are added to the underlying tools. Thoughts?

[phillebaba]

@seaneagan an issue with running docker-in-docker plugins would be dealing with the comparability issues as more and more k8s providers mover towards containerd. Personally i dont see an issue with allowing end users to extend the existing kustomzie-controller image with additional binaries, as a method of supporting plugins.

[polivbr]

I've put together a proof of concept patch that enables plugins by specifying an optional plugin home directory for a Kustomization. It would be great if we could get something like this merged in.

[fentas]

Any update on this? :+1:

[ringerc]

The outfit I've joined is running an elderly fork of the k8s controller to support --enable-alpha-plugins behavior - so yes, very very much so.