terraform-provider-kubernetes icon indicating copy to clipboard operation
terraform-provider-kubernetes copied to clipboard

Resource `kubernetes_node_taint` incorrectly denies the creation of multiple taints that share the same key but different effects

Open xinkechen-evernorth opened this issue 11 months ago • 1 comments

Terraform Version, Provider Version and Kubernetes Version

Terraform version: v1.10.3
Kubernetes provider version: v2.35.1
Kubernetes version: 1.31.2 (KinD)

Affected Resource(s)

  • kubernetes_node_taint

Terraform Configuration Files

# Minimum reproducible example with KinD or any cluster
terraform {
  required_providers {
    kubernetes = {
      source = "hashicorp/kubernetes"
    }
  }
}


data "kubernetes_nodes" "cluster_nodes" {}

resource "kubernetes_node_taint" "multiple_effects_same_key" {
  metadata {
    name = data.kubernetes_nodes.cluster_nodes.nodes[0].metadata[0].name
  }

  taint {
    key    = "node-role.kubernetes.io/ingress"
    value  = "reserved"
    effect = "NoSchedule"
  }

  taint {
    key    = "node-role.kubernetes.io/ingress"
    value  = "reserved"
    effect = "NoExecute"
  }
}

Debug Output

https://gist.github.com/xinkechen-evernorth/4e72460bbfc50510915a497506ca4291

Panic Output

N/A

Steps to Reproduce

From personal research, this behavior of allowing taints over the same key but different effects has been part of K8S since about 2016 (see upstream PR) and should be applicable for all versions since then.

As another example, the docs for a popular Kubernetes distribution, OKD/Red Hat OpenShift, also has the explicit instruction to users to create dedicated "infrastructure" nodes of the following specification:

spec:
  taints:
    - key: node-role.kubernetes.io/infra
      effect: NoSchedule
      value: reserved
    - key: node-role.kubernetes.io/infra
      effect: NoExecute
      value: reserved

References

  • #2611 - my proposed change to extend the resource to support taints using the same key but with different effects

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

xinkechen-evernorth avatar Jan 05 '25 20:01 xinkechen-evernorth

  • This feature is required since current version of kubernetes_node_taint resource does not work properly with samekey, value but has a differet effect.
  • eg) I have two kubernetes_node_taint like below:
resource "kubernetes_node_taint" "noschedule" {
  for_each = toset(local.node_names)
  # force = true
  metadata {
    name = each.key
  }
  taint {
    key    = "app"
    value  = "test"
    effect = "NoSchedule"
  }
}
resource "kubernetes_node_taint" "noexecute" {
  for_each = toset(local.node_names)
  # force = true
  metadata {
    name = each.key
  }
  taint {
    key    = "app"
    value  = "test"
    effect = "NoExecute"
  }
}
  • Above code must taint the worker node like this:
Taints:             app=test:NoExecute
                    app=test:NoSchedule
  • But actual output is just one taint applied:
Taints:         app=test:NoExecute

or

Taints:         app=test:NoSchedule
  • please refer the attached screenshot:

Image

luckymagic7 avatar Feb 14 '25 12:02 luckymagic7