ignore_annotations in provider configuration is not respected by kubernetes_manifest resource

[cwstrommer]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 1.8.5
Kubernetes provider version: 2.35.1
Kubernetes version: 1.31.5

Affected Resource(s)

• kubernetes_manifest

Terraform Configuration Files

resource "kubernetes_manifest" "argo_application_crd" {
  computed_fields = [
      "metadata.finalizers",
      "metadata.labels",
      "operation.initiatedBy.automated",
      "operation.initiatedBy.username",
      "operation.retry.backoff.duration",
      "operation.retry.backoff.factor",
      "operation.retry.backoff.maxDuration",
      "operation.retry.limit",
      "operation.sync.prune",
      "operation.sync.resources",
      "operation.sync.results",
      "operation.sync.revision",
      "operation.sync.syncOptions",
      "spec.ignoreDifferences",
      "status", 
  ]
  
  manifest = {
    "apiVersion" = "argoproj.io/v1alpha1"
    "kind"       = "Application"
    "metadata"   = {
      "name" = "application",
      "namespace" = "argocd",
      "annotations" = {
        "notifications.argoproj.io/subscribe.on-activity.slack" : ""
      }
    }
    "spec" = {
      "destination" = {  
        "namespace" = "default"
        "server"    = "https://kubernetes.default.svc"
      }
      "project" = "default"
      "source" = {
        "helm" = {
          "releaseName" = "test"
          "valueFiles" = ["values.yaml"]
        "path"           = "foo"
        "repoURL"        = <redacted>
        "targetRevision" = "main"
      }
    }
  }
  
  field_manager {
    force_conflicts = true
  }
}

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.31"
    }
  }
}  

provider "kubernetes" {
  host  = "https://localhost:6443"

  ignore_annotations = [    
    "notified.notifications.argoproj.io",
  ]
}

Steps to Reproduce

1. terraform apply

Expected Behavior

• Terraform applies the manifest.
• The ArgoCD controller sees the notifications.argoproj.io/subscribe... annotation, sends a message, and adds the notified.notifications.argoproj.io annotation in return to mark the notification state of the application.
• Terraform ignores the notified.notifications.argoproj.io annotation both after the current apply and in future plans.

Actual Behavior

• Terraform applies the manifest.
• The ArgoCD controller sees the notifications.argoproj.io/subscribe... annotation, sends a message, and adds the notified.notifications.argoproj.io annotation in return to mark the notification state of the application.
• Terraform errors out with

Error: Provider produced inconsistent result after apply

When applying changes to kubernetes_manifest.argo_application_crd, provider
"provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an
unexpected new value: .object.metadata.annotations: new element
"notified.notifications.argoproj.io" has appeared.

This is a bug in the provider, which should be reported in the provider's own issue tracker.

On retry, terraform attempts to remove the annotation, and after apply returns the same error because the controller has added the notification back.

Important Factoids

• We need to manage some annotations on the resource to manage the ArgoCD controller behaviour w.r.t the application resource, so we cannot use metadata.annotations in computed_fields.
• Everything else works as expected if we leave the notifications.argoproj.io/subscribe out, because then ArgoCD will not immediately modify the application.
• I tested variations of the ingore_annotations entry such as "notified\\.notifications\\.argoproj\\.io" and ".*argoproj\\.io.*" with no difference in results.

References

https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs#ignore-kubernetes-annotations-and-labels

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[cwstrommer]

PS: I'm currently partly working around the issue by keeping metadata.annotations in computed_fields, and then managing the annotations "out of band" with

resource "kubernetes_annotations" "argocd_application" {
  api_version = "argoproj.io/v1alpha1"
  kind        = "Application"
  metadata {
    name      = "application"
    namespace = "argocd"
  }
  "annotations" = {
    "notifications.argoproj.io/subscribe.on-activity.slack" : ""
  }
  field_manager = "Terraform-annotations"

  depends_on = [kubernetes_manifest.argo_application_crd]
}

This however is just a hack - it requires a separate field manager string (otherwise the kubernetes_annotations will attempt to clobber the application spec fields), and means I cannot reliably use the wait field of the kubernetes_manifest resource because not all the annotations may be correct at the time.