terraform-provider-kubernetes
terraform-provider-kubernetes copied to clipboard
hashicorp
Issues with null values and kubernetes_manifest for CRDs
Terraform version, Kubernetes provider version and Kubernetes version
Terraform version: Terraform v1.9.1
Kubernetes Provider version: 2.35.1
Kubernetes version: testing using kind locally, kind version 0.26.0 and in GKE , 1.30.5-gke.1443001
Question
Setup and Observed Behavior
We are developing a CRD and I am having trouble aligning behavior between the CRD definition and terraform and getting kubernetes and terraform to play nicely with each other. All my problems right now are around optional/nullable values in the CRD.
Abbreviated spec, showing a couple of fields in question;
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
name: "testing"
spec:
group: "testing"
names:
kind: "..."
scope: "Namespaced"
versions:
- name: "v2alpha1"
schema:
openAPIV3Schema:
properties:
spec:
properties:
deployment:
properties:
labels:
additionalProperties:
type: "string"
nullable: true
type: "object"
probes:
nullable: true
properties:
defaultProbe:
nullable: true
properties:
failureThreshold:
type: "integer"
initialDelaySeconds:
type: "integer"
path:
type: "string"
periodSeconds:
type: "integer"
port:
type: "integer"
timeoutSeconds:
type: "integer"
required:
- "failureThreshold"
- "initialDelaySeconds"
- "path"
- "periodSeconds"
- "port"
- "timeoutSeconds"
type: "object"
livenessProbe:
nullable: true
properties:
failureThreshold:
type: "integer"
initialDelaySeconds:
type: "integer"
path:
type: "string"
periodSeconds:
type: "integer"
port:
type: "integer"
timeoutSeconds:
type: "integer"
required:
- "failureThreshold"
- "initialDelaySeconds"
- "path"
- "periodSeconds"
- "port"
- "timeoutSeconds"
type: "object"
readinessProbe:
nullable: true
properties:
failureThreshold:
type: "integer"
initialDelaySeconds:
type: "integer"
path:
type: "string"
periodSeconds:
type: "integer"
port:
type: "integer"
timeoutSeconds:
type: "integer"
required:
- "failureThreshold"
- "initialDelaySeconds"
- "path"
- "periodSeconds"
- "port"
- "timeoutSeconds"
type: "object"
startupProbe:
nullable: true
properties:
failureThreshold:
type: "integer"
initialDelaySeconds:
type: "integer"
path:
type: "string"
periodSeconds:
type: "integer"
port:
type: "integer"
timeoutSeconds:
type: "integer"
required:
- "failureThreshold"
- "initialDelaySeconds"
- "path"
- "periodSeconds"
- "port"
- "timeoutSeconds"
type: "object"
type: "object"
status:
properties:
...
served: true
storage: true
subresources:
status: {}
If we look at the probes value specifically, everything is optional, and the individual probes define required fields for when they are specified.
If I try to apply a state like the following:
resource "kubernetes_manifest" "echo-service" {
manifest = {
# ...
}
"spec" = {
# ..
"deployment" = {
probes = {
defaultProbe = null
}
# ...
}
}
}
I get errors like the following, for every field that is a part of probe definition;
│ Error: spec.deployment.probes.defaultProbe.port
│
│ with kubernetes_manifest.echo-service,
│ on main.tf line 58, in resource "kubernetes_manifest" "echo-service":
│ 58: resource "kubernetes_manifest" "echo-service" {
│
│ Required value
If I just exclude probes altogether it works as expected, and it also doesn't error on the missing values, like livenessProbe, just on the defaultProbe.
Not including the value altogether is fine, except for when I try to create a module that defines the service for easier use in terraform.
If I define my variable in the module like:
variable "services" {
type = map(object({
name = string
deployment = object({
...
probes = optional(object({
defaultProbe = optional(object({
path = string
port = number
periodSeconds = number
initialDelaySeconds = number
timeoutSeconds = number
failureThreshold = number
}))
livenessProbe = optional(object({
path = string
port = number
periodSeconds = number
initialDelaySeconds = number
timeoutSeconds = number
failureThreshold = number
}))
startupProbe = optional(object({
path = string
port = number
periodSeconds = number
initialDelaySeconds = number
timeoutSeconds = number
failureThreshold = number
}))
readinessProbe = optional(object({
path = string
port = number
periodSeconds = number
initialDelaySeconds = number
timeoutSeconds = number
failureThreshold = number
}))
}))
})
}))
}
All those probe values end up always set to null and passed into the resource, which then causes the error above for every field in Probes.
Another issue I'm seeing with nulls is with the labels. If I set labels = null, every time I run a plan it shows as unknown;
# kubernetes_manifest.echo-service will be updated in-place
~ resource "kubernetes_manifest" "echo-service" {
~ object = {
~ spec = {
~ deployment = {
+ labels = (known after apply)
# (9 unchanged attributes hidden)
}
# (7 unchanged attributes hidden)
}
# (3 unchanged attributes hidden)
}
# (1 unchanged attribute hidden)
# (1 unchanged block hidden)
}
No matter how many plan/applys I run, with no changes, I always get the above. Oddly, if I set the top level probes value to null, it doesn't exhibit this behavior. It shows no difference with that value between runs.
Actual Question Is there a way to treat null values the same as if they weren't set when dealing with a kubernetes_manifest? Or some other way in TF itself to acheive what we are after here? Namely, being able to define a module for our CRD to apply validation in TF and for helping others consume the CRD.
The module/object behavior seems like a fundamental concept in Terraform which I understand is likely not going to change. Ideally, optional fields are truly optional. If they are not passed they are not set, just like if I didn't set them when writing the manifest directly. I guess if there as no difference in behavior between null and not-set, it wouldn't be a problem.
If I can provide any more output or information to help someone help me understand what's going on here please let me know. I can see if I can come up with a smaller test case, but not sure how quickly I'll be able to produce that.
