Old labels remain in PrometheusRules rule specifications

[NBardelot]

Terraform Version, Provider Version and Kubernetes Version

• Terraform version: 1.6.4
• Kubernetes provider version: hashicorp/kubernetes v2.27.0
• Kubernetes version: 1.29

Affected Resource(s)

• API = monitoring.coreos.com/v1
• kind = PrometheusRule

Steps to Reproduce

1. Create a PrometheusRule by specifiying a manifest as HCL code (ressource "kubernetes_manifest"), including a set of labels for a rule.
2. Remove one of the labels.
3. Apply once again.
4. See that the removed label is present in the Terraform state (value now being null), and is still present in the effective PrometheusRule in Kubernetes with its old value.

Trying to delete/recreate the ressource would produce such logs in Terraform's output:

kubernetes_manifest.prometheus_rules: Creating...
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to kubernetes_manifest.prometheus_rules, provider
│ "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an
│ unexpected new value: .object.spec.groups[0].rules[0].labels: element
│ "opsgenie" has vanished.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.
╵

With the plan beforehand indicating:

kubernetes_manifest.prometheus_rules is tainted, so must be replaced.

Expected Behavior

The label should be removed from the tfstate, and from the ressource in Kubernetes.

Important Factoids

This might not be specific to PrometheusRule, but might happen for other manifests.

[BBBmau]

Hello, thanks for opening this issue @NBardelot. Can you include a simple TF config that can reproduce this error output. This could be related to #2436 since I see that labels is a map[string] type. Being able to have the config will confirm whether it's related or not.

[NBardelot]

The terraform module would look like this:

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.15.0"
    }
  }
}

variable "k8s_cluster" {
  type        = string
  description = "kubernetes cluster"
}

variable "prometheus_rules_spec" {
  type = object({
    groups = list(object({
      name = string
      rules = list(object({
        alert       = string
        annotations = map(string)
        expr        = string
        labels = object({
          severity = string
        })
      }))
    }))
  })
}

provider "kubernetes" {
  config_context = var.k8s_cluster
  config_path    = "~/.kube/config"
}

resource "kubernetes_manifest" "prometheus_rules" {
  manifest = {
    apiVersion = "monitoring.coreos.com/v1"
    kind       = "PrometheusRule"
    metadata = {
      name      = "example"
      namespace = "somewhere"
    }
    spec = {
      groups = [
        for group in var.prometheus_rules_spec.groups : {
          name = group.name
          rules = [
            for rule in group.rules : {
              alert       = rule.alert
              annotations = rule.annotations
              expr        = rule.expr
              labels = merge(
                rule.labels,
                {
                  namespace = "somewhere"
                  severity  = rule.labels.severity
                }
              )
            }
          ]
        }
      ]
    }
  }
}

And the tfvars file:

prometheus_rules_spec = {
  groups = [
    {
      name = "group1"
      rules = [
        {
          alert = "alert1"
          annotations = {
            description = "example"
            summary     = "example"
          }
          expr = "1 != 1"
          labels = {
            severity = "warning"
          }
        }
      ]
    }
  ]
}

[NBardelot]

I've found a workaround with this - cleaner - implementation:

terraform {
  required_version = ">= 1.6.0"
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.15.0"
    }
  }
}

variable "k8s_cluster" {
  type        = string
  description = "kubernetes cluster"
}

variable "prometheus_rules_spec" {
  type = object({
    groups = list(object({
      name = string
      rules = list(object({
        alert       = string
        annotations = map(string)
        expr        = string
        severity_label    = string
        additional_labels = optional(map(string))
      }))
    }))
  })
}

provider "kubernetes" {
  config_context = var.k8s_cluster
  config_path    = "~/.kube/config"
}

resource "kubernetes_manifest" "prometheus_rules" {
  manifest = {
    apiVersion = "monitoring.coreos.com/v1"
    kind       = "PrometheusRule"
    metadata = {
      name      = "example"
      namespace = "somewhere"
    }
    spec = {
      groups = [
        for group in var.prometheus_rules_spec.groups : {
          name = group.name
          rules = [
            for rule in group.rules : {
              alert       = rule.alert
              annotations = rule.annotations
              expr        = rule.expr
              labels = merge(
                rule.additional_labels,
                {
                  namespace = "somewhere"
                  severity  = rule.severity_label
                }
              )
            }
          ]
        }
      ]
    }
  }
}

And the tfvars file:

prometheus_rules_spec = {
  groups = [
    {
      name = "group1"
      rules = [
        {
          alert = "alert1"
          annotations = {
            description = "example"
            summary     = "example"
          }
          expr = "1 != 1"
          severity_label = "warning"
          additional_labels = {
            opsgenie     = "test"
          }
        }
      ]
    }
  ]
}