Ingress_v1 deletes "internal" (but not really) annotations

[joey-squid]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: v1.10.4
Kubernetes provider version: v2.35.1
Kubernetes version: v1.31.4-eks-2d5f260

Affected Resource(s)

• kubernetes_ingress_v1

Terraform Configuration Files

provider "kubernetes" {
  config_path    = "~/.kube/config"
  config_context = "sandbox-use1-default"
}

resource "kubernetes_ingress_v1" "joey" {
  metadata {
    name = "joey"

    annotations = {
      foo = "bar"
    }
  }
  spec {
    default_backend {
      service {
        name = "joey"
        port {
          number = 8000
        }
      }
    }
  }
}

Debug Output

https://gist.github.com/joey-squid/41b5be220ee543dc3c93541c058496b7

Panic Output

Steps to Reproduce

1. terraform apply
2. kubectl annotate ingress joey foo.kubernetes.io=bar
3. (kubectl describe ingress joey shows both annotations)
4. Update the spec, e.g. to change the annotation to foo = "bar1"
5. terraform apply. Note that there is no mention of foo.kubernetes.io in the plan
6. kubectl describe ingress joey. The annotation is gone!

Expected Behavior

Terraform should not silently delete annotations that it ignores, and if it does they should be in the plan.

Actual Behavior

Terraform deleted those annotations.

Important Factoids

The Terraform k8s provider considers all annotations ending in .kubernetes.io to be internal (with a few exceptions), despite the fact that some services outside of k8s proper use annotations ending in .kubernetes.io. In my case, these are alb.ingress.kubernetes.io/* annotations, though there is also external DNS.

I believe the same issue would happen if my annotation were in the ignore_annotations provider-level parameter, though I haven't tested this.

References

I have opened issue #2666 about not ignoring kubernetes.io annotations by default at all.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[joey-squid]

Perhaps missing from my report is the fact that this does not apply to all k8s resource types. I tried reproducing this with a pod instead of an ingress and it didn't work. I assume that annotations are munged differently on the k8s backend.