create module: CRD and default CR validation

[clebs]

Description

Kyma modules are represented at runtime with a specific CRD which needs to conform to the lifecycle-manager's specification. To ensure a valid CRD is defined for a module it needs to be validated.

Additionally, to instantiate a module, developers have to provide a default CR instance for the module they want to create a module template. Currently the CR is not validated before building the module and could later fail when deploying it to a cluster.

To make module development safer we have to validate the CRD and the default CR against the CRD usually found in the module's operator.

AC

• Validate that the CRD defined in the module is correct as per lifecycle-manager.
• Validate that the default CR provided for the module template is correct.

Reasons

Avoid building defective modules and templates.

[jakobmoellerdev]

Currently known ACs:

• [ ] Verify status.state to be conformant with lifecycle-manager

[Tomasz-Smelcerz-SAP]

The kubectl tool offers client-side validation of the CR (vs CRD schema), but it reads the CRD OpenAPI v3 schema from the cluster. Perhaps we could reuse that code, pointing to a definition from a local file instead. We can't query the cluster for the CRD, because at the development time, there may be no cluster at all. In addition kubectl codebase contains all the k8s ecosystem specific patches already. We could use a generic OpenAPI v3 schema validator, but then we risk that some custom validations are not implemented there. Besides that, there isn't a lot tools for OpenAPI v3 schema validation. I was able to find this list, just one generic project written in golang.

[jakobmoellerdev]

After looking at this again:

1. The CR cannot be safely verified against an OpenAPI scheme unless we do client side validation against the CRD bundled in the module
2. To find the correct CRD to verify against we have to iterate over all CRDs bundled in the module and compare it through the GVK against one specified in default.yaml
3. In addition to normal YAML validation against the CR I think the more important part of validation would be to validate against status.state, maybe we should prioritize this first because the webhook in lifecycle-manager already contains such a validation

[Tomasz-Smelcerz-SAP]

@jakobmoellersap status object shouldn't be set by end users, but by the controller only, isn't it? The scope of the validation, as I understand it, is to detect errors in the user's input, i.e. CR spec object. I'd say we should perhaps ensure that there's no status object at all in the CR's definition YAML. The proper status will be set/created once the resource is reconciled by the controller in the target cluster.

[clebs]

Hi @Tomasz-Smelcerz-SAP, @jakobmoellersap,

I clarified the AC and edited the title and description to reflect the dual validation needed to have safe module development.

[Tomasz-Smelcerz-SAP]

I have tried to debug how the kubectl tool is performing the CR validation, but I haven't managed to actually debug the code within one day - it's too complex and too much dependent on the interaction with the api-server. I think that this project is the way to go - at least from what I saw it offers very simple API and there are nice test cases to learn the usage from.

@clebs Any hints on how to verify if the schema itself (CRD) conforms to the Lifecycle-Manager contract? There is no "schema for schema". We can try to do it programmatically somehow, but I'm afraid it would be fragile.

[clebs]

Hi @Tomasz-Smelcerz-SAP, for the CRD itself we will have to define certain rules manually and check for them programatically. Best is to align with the module and lifecycle managers to see which properties need to be validated.

[Tomasz-Smelcerz-SAP]

To see how hard it is to actually validate CRDs exactly like K8s does: https://danielmangum.com/posts/how-kubernetes-validates-custom-resources/

[Tomasz-Smelcerz-SAP]

We decided to use EnvTest library for testing the CR. This gives us the most complete and correct validation, because it's performed by the locally running K8s api-server. It will also validate the CRD automatically - if there's any error in the CRD definition, testenv will reject it. The downside of this approach is the time it takes to bootstrap the EnvTest. It takes over 10 seconds on average on my machine.

[Tomasz-Smelcerz-SAP]

@clebs How do we find a module CRD? The "default.yaml" is in the module root, and has a known name. The CRD file is embedded in the "charts" - somewhere. Do we have any contract on the name/placement of this file? Can there be more that one CRD there?

[clebs]

Hi @Tomasz-Smelcerz-SAP, as you guessed, the CRD is on the operator and there should be only one which matches the operator name. Can you confirm the premise @jakobmoellersap?