terraform-provider-kubernetes-alpha
terraform-provider-kubernetes-alpha copied to clipboard
Installing manifests that depend on CRDs created in previous resources. Plan fails as not yet present in cluster and no way to apply without targeting and using old provider.
Terraform, Provider, Kubernetes versions
Terraform version: v0.14.11
Provider version: v0.3.3
Kubernetes version: 1.18.16-gke.2100
Affected Resource(s)
Terraform Configuration Files
resource "helm_release" "istio_operator" {
name = "istio-operator"
chart = "/data/shared/istio-1.9.3/manifests/charts/istio-operator"
namespace = var.istio_namespace
set {
name = "hub"
value = var.release_hub
}
set {
name = "operatorNamespace"
value = var.operator_namespace
}
set {
name = "watchedNamespaces"
value = var.istio_namespace
}
}
resource "time_sleep" "destroy_istio_resources_before_operator" {
depends_on = [helm_release.istio_operator]
destroy_duration = "60s"
}
resource "kubernetes_manifest" "istio" {
provider = kubernetes-alpha
manifest = yamldecode(templatefile(var.istio_tpl_yml_path_override != null ? var.istio_tpl_yml_path_override : "${path.module}/${var.istio_tpl_yml_path}", {
istio_namespace = var.istio_namespace
replicas = var.control_plane_replicas
release_hub = var.release_hub
release_tag = var.release_tag
priority_class_name = var.priority_class_name
node_pool_affinity = var.node_pool_affinity
tolerations = var.tolerations
resources = var.resources
access_log_format_override = var.access_log_format_override
remove_node_affinity = var.remove_node_affinity
remove_tolerations = var.remove_tolerations
}))
wait_for = {
fields = {
"status.status" = "HEALTHY"
}
}
depends_on = [helm_release.istio_operator, time_sleep.destroy_istio_resources_before_operator]
}
Debug Output
Panic Output
Steps to Reproduce
Expected Behavior
What should have happened? Be able to plan/apply by honouring depends_on and waiting for each resource to be applied in turn.
Installing the istio_operator adds the install.istio.io
group, in turn, when that is available, installing istio will add a few more such as security.istio.io
Actual Behavior
What actually happened?
The provider doesn't know how to plan the resources as their dependencies have not yet been created.
Error: rpc error: code = Unknown desc = no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
Error: Failed to determine GroupVersionResource for manifest
on .terraform/modules/istio/istio-control-plane.tf line 58, in resource "kubernetes_manifest" "istio":
58: resource "kubernetes_manifest" "istio" {
no matches for kind "IstioOperator" in group "install.istio.io"
To get around it:
- pin down the kubernetes-alpha version to 0.2.1
- terraform apply -target helm_release.istio_operator
- wait a bit
- terrafomr apply -target kubernetes_manifest.istio
- wait a bit
- apply the rest of the state
IF running 0.3.3 provider version then we get:
- terraform apply -target helm_release.istio_operator
- wait a bit
- error below:
Error: rpc error: code = Unknown desc = no matches for kind "AuthorizationPolicy" in version "security.istio.io/v1beta1"
Error: No valid OpenAPI definition
on .terraform/modules/istio/istio-control-plane.tf line 58, in resource "kubernetes_manifest" "istio":
58: resource "kubernetes_manifest" "istio" {
Resource IstioOperator.v1alpha1.install.istio.io does not have a valid OpenAPI
definition in this cluster.
Usually this is caused by a CustomResource without a schema.
which is mentioned here: https://github.com/hashicorp/terraform-provider-kubernetes-alpha/issues/181#issuecomment-830264484
$ kubectl api-resources | grep IstioOperator
istiooperators iop install.istio.io true IstioOperator
which means we cannot even target the istio install with latest provider.
Any ideas are welcome as currently we have no way of applying the state cleanly.
Thanks
Important Factoids
References
- GH-1234
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
I also hace such error. In my case i have such kubernetes_manifest resource
resource "kubernetes_manifest" "vertical-pod-autoscaler" {
provider = kubernetes-alpha
manifest = {
"apiVersion" = "autoscaling.k8s.io/v1"
"kind" = "VerticalPodAutoscaler"
...
Terraform plan fails with error
╷
│ Error: No valid OpenAPI definition
│
│ with kubernetes_manifest.vertical-pod-autoscaler,
│ on vpa.tf line 4, in resource "kubernetes_manifest" "vertical-pod-autoscaler":
│ 4: resource "kubernetes_manifest" "vertical-pod-autoscaler" {
│
│ Resource VerticalPodAutoscaler.v1.autoscaling.k8s.io does not have a valid OpenAPI definition in this cluster.
│
│ Usually this is caused by a CustomResource without a schema.
Cluster has verticalpodautoscalers api resource with Kind: VerticalPodAutoscaler but with NAME verticalpodautoscalers
kubectl api-resources --api-group=autoscaling.k8s.io
NAME SHORTNAMES APIVERSION NAMESPACED KIND
verticalpodautoscalers vpa autoscaling.k8s.io/v1 true VerticalPodAutoscaler
It looks like kubernetes-alpha checks Kind instead of CRD. In my exaple it checks Resource VerticalPodAutoscaler.v1.autoscaling.k8s.io instead of Resource VerticalPodAutoscalers.v1.autoscaling.k8s.io
You should use api resource name instead of kind for validating OpenAPI definition