Use cdk8s library for generating Kubernetes yamls

[girishc13]

Describe the feature

The current k8s yaml generation relies on templating static k8s yaml files. The static files don't support k8s version specific resource definition and migration to newer versions. The cdk8s library provides features to generate k8s yaml files programmatically and target k8s version specific resources. The library supports official k8s objects and custom resources if needed.

Your proposal

Create a custom App python class to represent a Flow and generate the dependent k8s resources using code which can be dumped into yamls. We can allow user specific customizations required for their target k8s version.

---

Environment

Screenshots

[JoanFM]

I like the idea, but not sure if we should put effort here if we will move towards the Operator world

[girishc13]

I think this will become more important when we move to the operator because the deployment yamls are still generated by the flow.to_kubernetes_yaml() method. AFAIU we need this conversion to convert the Flow topology into k8s yamls.

[JoanFM]

I am not sure this will be the case in Operator @deepankarm ??? If it is the case, then definitely this would make sense.

[deepankarm]

Yes. Operator still generates the K8S YAMLs from Flow YAML using to_kubernetes_yaml() provided by core. We had this discussion before and decided to move this conversion from Flow to K8S yamls in v2 of operator.

I don't know anything about cdk8s. @girishc13 The JCloud API still receives the Flow YAML itself. How does Flow YAML to Golang based K8S objects conversion happens if we use cdk8s?

[girishc13]

From my understanding, to create k8s resources we need to generate k8s manifest yaml which is currently done by the to_kubernetes_yaml() method. This is currently using static yaml files rather than using compliant k8s api schemas. This feature will affect the to_kubernetes_yaml() method that will use compliant schemas to generate the k8s yamls.

The other alternative is to parse the Flow YAML by the operator which will operate directly on the k8s api schemas. I'm not aware if we are planning to add this feature to the operator.

Also there might be other uses who might want to manage and customise the k8s resources manually without the operator. For them generating a compliant k8s manifests will be important.

[JoanFM]

Yes it may be interesting.

I propose to revisit this when we go for the v2 of Operator which should not take soo long to go after