feature: ability to manage CRDs in KCP using terraform

[andrewlee1089]

Feature Description

I can't quite decide whether to raise this with the KCP community or the terraform-provider-kubernetes community. I think you can argue it either way!

The problem is that using if you use custom resources in a KCP, then you cannot use terraform to manage them:

 terraform apply
╷
│ Error: Plugin error
│
│   with kubernetes_manifest.mycustomkind,
│   on main.tf line 15, in resource "kubernetes_manifest" "mycustomkind":
│   15: resource "kubernetes_manifest" "mycustomkind" {
│
│ The plugin returned an unexpected error from plugin.(*GRPCProvider).PlanResourceChange: rpc error: code = Unknown desc = failed to determine resource type ID: cannot get resource type from OpenAPI
│ (mycustomapi.com/v1alpha1, Kind=mycustomkind): mycustomapi.com/v1alpha1, Kind=mycustomkind resource not found in OpenAPI index

This is because the terraform code in TFTypeFromOpenAPI (https://github.com/hashicorp/terraform-provider-kubernetes/blob/main/manifest/provider/resource.go#L85C30-L85C47) assumes that custom resource types:

• will have a CRD
• will have an OpenAPISchema on that CRD

KCP obviously breaks these assumptions. The OpenAPISchema is still there - on the APIResourceSchema object.

At this point I'm just raising this without any expectation it will be fixed - but I am interested in finding out if other KCP users would also like this feature.

Proposed Solution

I think it's incumbent on the KCP community to push a fix to the terraform-provider-kubernetes to cope with custom resources where is no CRD.

Alternative Solutions

No response

Want to contribute?

• [ ] I would like to work on this issue.

Additional Context

No response

[embik]

Hey @andrewlee1089 - thanks for using kcp and reporting this issue! As you mentioned, this seems to be an issue with the Terraform provider, we can't do much about it. I'm happy to keep this open, should we maybe convert it into a discussion?

The code in the provider isn't fully clear to me, but it seems to me that ideally, the provider would look for a CRD and if it cannot find one, fall back to OpenAPI discovery directly. I wonder if this is something that could be raised upstream?

[xrstf]

Just to be sure, is your Kind really all lowercase?

[yhrn]

I have looked into this and I believe that the reason why terraform-provider-kubernetes fails is not exactly as described in this issue. While it is true that it looks for CRDs and extracts OpenAPI schema from there, it will always fall back to the /openapi/v2 endpoint. As pointed out in #1818 (closed becasue of inactivity), KCP does not serve OpenAPI v2 specifications for user defined types.

terraform-provider-kubernetes tries to get the schema from CRDs first becasue they contain OpenAPI v3 schema (see this comment that explains why this is important) and /openapi/v3 only went GA in k8s v1.27.

So, while fixing #1818 would be nice, it would still be a bit risky to use the terraform-provider-kubernetes in its current state against KCP because it would do a, potentially lossy, translation from OpenAPI v2 to v3. The "right" way to get this to work seems to be to get terraform-provider-kubernetes to use /openapi/v3 for all resources if the server is v1.27+ (should work for newer KCP versions thanks to #3242) and only fall back to the current behavior for older versions.