terraform-provider-kubernetes
terraform-provider-kubernetes copied to clipboard
Status info missing from kubernetes_manifest object attribute after apply
Terraform Version, Provider Version and Kubernetes Version
Terraform v1.1.4
on darwin_arm64
+ provider registry.terraform.io/hashicorp/google v4.10.0
+ provider registry.terraform.io/hashicorp/helm v2.4.1
+ provider registry.terraform.io/hashicorp/kubernetes v2.8.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
Kubernetes version 1.21
Affected Resource(s)
- kubernetes_manifest
Terraform Configuration Files
resource "kubernetes_manifest" "prometheus_server_serviceattachment" {
count = var.psc_enabled ? 1 : 0
manifest = {
apiVersion = "networking.gke.io/v1beta1"
kind = "ServiceAttachment"
metadata = {
name = "${var.release_name}-server"
namespace = kubernetes_namespace.this.id
}
spec = {
connectionPreference = "ACCEPT_AUTOMATIC"
natSubnets = [
google_compute_subnetwork.prometheus_server_psc_subnet[0].name
]
proxyProtocol = false
resourceRef = {
kind = "Service"
name = kubernetes_service.prometheus_server_lb[0].metadata[0].name
}
}
}
wait_for = {
fields = {
"status.serviceAttachmentURL" = "^https:\\/\\/www.googleapis.com\\/.*\\/serviceAttachments\\/.*$"
}
}
}
Debug Output
$ tg console
> kubernetes_manifest.prometheus_server_serviceattachment[0].object
{
"apiVersion" = "networking.gke.io/v1beta1"
"kind" = "ServiceAttachment"
"metadata" = {
"annotations" = tomap(null) /* of string */
"clusterName" = tostring(null)
"creationTimestamp" = tostring(null)
"deletionGracePeriodSeconds" = tonumber(null)
"deletionTimestamp" = tostring(null)
"finalizers" = tolist([
"networking.gke.io/service-attachment-finalizer",
])
"generateName" = tostring(null)
"generation" = tonumber(null)
"labels" = tomap(null) /* of string */
"managedFields" = null /* tuple */
"name" = "prometheus-server"
"namespace" = "prometheus"
"ownerReferences" = tolist(null) /* of object */
"resourceVersion" = tostring(null)
"selfLink" = tostring(null)
"uid" = tostring(null)
}
"spec" = {
"connectionPreference" = "ACCEPT_AUTOMATIC"
"consumerAllowList" = tolist(null) /* of object */
"consumerRejectList" = tolist(null) /* of string */
"natSubnets" = tolist([
"prometheus-server-psc-subnet-5c4bdd",
])
"proxyProtocol" = false
"resourceRef" = {
"apiGroup" = tostring(null)
"kind" = "Service"
"name" = "prometheus-server-lb"
}
}
}
Expected Behavior
The documentation states "the object
attribute contains the state of the resource as returned by the Kubernetes API, including all default values."
Various objects will expose valuable information in their status. In this case, I'd like to be able to have the Terraform module export the service attachment URL as an output:
output "service_attachment_url" {
value = kubernetes_manifest.prometheus_server_serviceattachment.object.status.serviceAttachmentURL
}
The output should contain the same value that is shown here when getting the object via kubectl:
$ kubectl -n prometheus get serviceattachment prometheus-server -o yaml
apiVersion: networking.gke.io/v1beta1
kind: ServiceAttachment
metadata:
creationTimestamp: "2022-02-15T17:55:11Z"
finalizers:
- networking.gke.io/service-attachment-finalizer
generation: 3
name: prometheus-server
namespace: prometheus
resourceVersion: "8019482"
uid: 64e16c47-26b5-4dd3-aff7-289fed47bc29
spec:
connectionPreference: ACCEPT_AUTOMATIC
natSubnets:
- prometheus-server-psc-subnet-5c4bdd
proxyProtocol: false
resourceRef:
kind: Service
name: prometheus-server-lb
status:
forwardingRuleURL: https://www.googleapis.com/compute/beta/projects/my-project/regions/us-west1/forwardingRules/ad3bccdeca9114142acdc91ddcb5b702
lastModifiedTimestamp: "2022-02-15T17:55:15Z"
serviceAttachmentURL: https://www.googleapis.com/compute/beta/projects/my-project/regions/us-west1/serviceAttachments/k8s1-sa-hl4gna7v-prometheus-prometheus-server-as5d6ghe
Actual Behavior
The resources is created, but there is no way to access the serviceAttachmentURL
value in Terraform.
Community Note
- Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
- If you are interested in working on this issue or have submitted a pull request, please leave a comment
Hi,
The "status" attribute is removed by the provider logic on purpose. This is becuase the value of status can change on the API side without intervention from Terraform and thus produce unactionable diffs in the following plan
operations.
The status attribute is better reflected through a data-source (read only) and this will be available once PR https://github.com/hashicorp/terraform-provider-kubernetes/pull/1548 will be released.
Let me know if this doesn't sufficiently clarify the issue.
the value of status can change on the API side without intervention from Terraform
Understand this, both status and annotations frequently change after initial resource creation, especially for objects which have custom controllers (Ingress and ServiceAttachement being two examples) that create resources within a given cloud provider's system to provide X functionality.
thus produce unactionable diffs in the following plan operations The status attribute is better reflected through a data-source (read only)
This is where I would beg to differ. The diff would be one of a state change, it should show up as a "change made outside of Terraform" and when tf apply
or tf apply -refresh-only
is run the diff should come in and be reflected in state.
One should not need to have a data resource that depends on a resource you just created to get a value from the created resource.
An existing example where this works as one would naturally have expected here is the kubernetes_ingress_v1
resource. You can specify wait_for_load_balancer = true
and it's creation will block until status[0].load_balancer
is no longer empty, allowing you to use the value the ingress controller added to the status object.
If you omit wait_for_load_balancer
so the resource creation doesn't wait, indicating completion as soon as the object is applied to k8s, the following happens:
a) Initial object is applied and the Ingress returned from the API looks like:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: gce
networking.gke.io/v1beta1.FrontendConfig: traefik
creationTimestamp: "2022-02-17T18:10:09Z"
finalizers:
- networking.gke.io/ingress-finalizer-V2
generation: 1
name: traefik
namespace: traefik
resourceVersion: "612524"
uid: 85d6bb25-8ed3-4389-8fc1-2d4c97df8e2c
spec:
defaultBackend:
service:
name: traefik
port:
name: web
tls:
- secretName: traefik-tls-algerdev-ptlabs-dev
status:
loadBalancer: {}
b) After some short amount of time — in this case, once the controller has created an http/s load balancer on Google Cloud — the object is updated by the controller and then looks like this:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/backends: '{"k8s1-9fd681d1-traefik-traefik-80-01cfbd6f":"Unknown"}'
ingress.kubernetes.io/forwarding-rule: k8s2-fr-hs6u9rbo-traefik-traefik-35kfb15b
ingress.kubernetes.io/https-forwarding-rule: k8s2-fs-hs6u9rbo-traefik-traefik-35kfb15b
ingress.kubernetes.io/https-target-proxy: k8s2-ts-hs6u9rbo-traefik-traefik-35kfb15b
ingress.kubernetes.io/redirect-url-map: k8s2-rm-hs6u9rbo-traefik-traefik-35kfb15b
ingress.kubernetes.io/ssl-cert: k8s2-cr-hs6u9rbo-kwnt3rdrvy2od6sz-66091e840b26166a
ingress.kubernetes.io/static-ip: k8s2-fr-hs6u9rbo-traefik-traefik-35kfb15b
ingress.kubernetes.io/target-proxy: k8s2-tp-hs6u9rbo-traefik-traefik-35kfb15b
ingress.kubernetes.io/url-map: k8s2-um-hs6u9rbo-traefik-traefik-35kfb15b
kubernetes.io/ingress.class: gce
networking.gke.io/v1beta1.FrontendConfig: traefik
creationTimestamp: "2022-02-17T18:10:09Z"
finalizers:
- networking.gke.io/ingress-finalizer-V2
generation: 1
name: traefik
namespace: traefik
resourceVersion: "613099"
uid: 85d6bb25-8ed3-4389-8fc1-2d4c97df8e2c
spec:
defaultBackend:
service:
name: traefik
port:
name: web
tls:
- secretName: traefik-tls-algerdev-ptlabs-dev
status:
loadBalancer:
ingress:
- ip: 34.149.125.208
c) Run tf apply
and you will see the change to the object's status
will show up as a change made outside of Terraform:
...
kubernetes_ingress_v1.traefik: Refreshing state... [id=traefik/traefik]
Note: Objects have changed outside of Terraform
Terraform detected the following changes made outside of Terraform since the
last "terraform apply":
# kubernetes_ingress_v1.traefik has changed
~ resource "kubernetes_ingress_v1" "traefik" {
id = "traefik/traefik"
~ status = [
~ {
~ load_balancer = [
~ {
~ ingress = [
+ {
+ hostname = ""
+ ip = "35.241.49.191"
},
]
},
]
},
]
~ metadata {
name = "traefik"
~ resource_version = "646816" -> "647519"
# (5 unchanged attributes hidden)
}
# (1 unchanged block hidden)
}
Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.
─────────────────────────────────────────────────────────────────────────────
No changes. Your infrastructure matches the configuration.
Your configuration already matches the changes detected above. If you'd like
to update the Terraform state to match, create and apply a refresh-only plan:
terraform apply -refresh-only
Releasing state lock. This may take a few moments...
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
The new value the controller added to status
is now stored in the Terraform state, even though the resource will never change it, and it's available to use via remote state, etc.
In the case of kubernetes_manifest
the wait_for
field exists, as can be seen in the example I originally posted, to block until this value is set by the controller which manages these objects on the cluster:
wait_for = {
fields = {
"status.serviceAttachmentURL" = "^https:\\/\\/www.googleapis.com\\/.*\\/serviceAttachments\\/.*$"
}
}
The examples in the documentation show configuration which basically does the same as wait_for_load_balancer
on the ingress resource:
# Check an ingress has an IP
"status.loadBalancer.ingress[0].ip" = "^(\\d+(\\.|$)){4}"
It stands to reason that if one is to block create and update calls until a field is set or has a particular value, that the value will be available on the object
when the call completes, and not require a data resource to fetch it. The data resource is valuable as well — don't get me wrong here — but it's use-case is slightly different in that it can be used to fetch information about something created outside the confines of the current Terraform module. It shouldn't be needed to gather info on something just created in the same module.
Hi,
The "status" attribute is removed by the provider logic on purpose. This is becuase the value of status can change on the API side without intervention from Terraform and thus produce unactionable diffs in the following
plan
operations. The status attribute is better reflected through a data-source (read only) and this will be available once PR #1548 will be released.Let me know if this doesn't sufficiently clarify the issue.
Hi, however in PR #1548, the status attribute is also deleted via RemoveServerSideFields
and unavailable via the data-source.
see #1699
Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!
This really needs to be addressed, not left to be closed.
We believe this issue has been resolved with PR https://github.com/hashicorp/terraform-provider-kubernetes/pull/1548 and PR https://github.com/hashicorp/terraform-provider-kubernetes/issues/1699. If this problem persists, please reopen a new issue and link back to this one. Thanks!