Rolebinding update fails

[szpuni]

Discovered edge case in Kubernetes provider when I try to update ClusterRoleBinding with count parameter. Idea here is that I have a ClusterRole with permissions to certain users. Then depending on environment (dev, prod etc) I create ClusterRoleBinding with name shared accross two resources controlled by count.

As shown in code if I set context variable to prod role is created and binding created with that role to a group. Now if I change context value to dev I get error Error: clusterrolebindings.rbac.authorization.k8s.io "my:developer" already exists

If I run terraform apply again, it will successfully run without issues. Cluster Role Bindings cannot be updated and needs to be replaced. Problem here is that provider in this edge case doesn't check metadata.name in resource for existence before it's trying to create new one. Looks like we are trying to create new resource with exact same name even tho it's existing in Kubernetes.

Terraform Version, Provider Version and Kubernetes Version

Terraform v1.0.4
on darwin_amd64
+ provider registry.terraform.io/hashicorp/aws v3.55.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.2.0
+ provider registry.terraform.io/hashicorp/helm v2.2.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.4.1
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/terraform-aws-modules/http v2.4.1

Affected Resource(s)

• kubernetes_cluster_role_binding

Terraform Configuration Files

resource "kubernetes_cluster_role" "developer" {
  count = var.context == "prod" ? 1 : 0
  metadata {
    name = "my:developer"
  }
  rule {
    api_groups = [""]
    verbs      = ["*"]
    resources = [
      "pods",
      "pods/log",
    ]
  }
  rule {
    api_groups = [
      "",
      "extensions",
      "apps",
      "batch"
    ]
    verbs = [
      "get",
      "list",
      "watch"
    ]
    resources = [
      "deployments",
      "events",
      "jobs",
      "cronjobs",
      "services",
      "namespaces",
      "namespaces/status"
    ]
  }
}

resource "kubernetes_cluster_role_binding" "developer-prod" {
  count = var.context == "prod" ? 1 : 0
  metadata {
    name = "my:developer"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = kubernetes_cluster_role.developer[0].metadata[0].name
  }
  subject {
    api_group = "rbac.authorization.k8s.io"
    kind      = "Group"
    name      = "my:developer"
  }
}

resource "kubernetes_cluster_role_binding" "developer" {
  count = var.context != "prod" ? 1 : 0
  metadata {
    name = "my:developer"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "cluster-admin"
  }
  subject {
    api_group = "rbac.authorization.k8s.io"
    kind      = "Group"
    name      = "my:developer"
  }
}

Debug Output

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  - destroy

Terraform will perform the following actions:

  # module.eks.module.eks-init.kubernetes_cluster_role.developer[0] will be destroyed
  - resource "kubernetes_cluster_role" "developer" {
      - id = "my:developer" -> null

      - metadata {
          - annotations      = {} -> null
          - generation       = 0 -> null
          - labels           = {} -> null
          - name             = "my:developer" -> null
          - resource_version = "841510" -> null
          - uid              = "a40b4f48-d7c3-4748-939c-fd257a7d29a7" -> null
        }

      - rule {
          - api_groups        = [
              - "",
            ] -> null
          - non_resource_urls = [] -> null
          - resource_names    = [] -> null
          - resources         = [
              - "pods",
              - "pods/log",
            ] -> null
          - verbs             = [
              - "*",
            ] -> null
        }
      - rule {
          - api_groups        = [
              - "",
              - "extensions",
              - "apps",
              - "batch",
            ] -> null
          - non_resource_urls = [] -> null
          - resource_names    = [] -> null
          - resources         = [
              - "deployments",
              - "events",
              - "jobs",
              - "cronjobs",
              - "services",
              - "namespaces",
              - "namespaces/status",
            ] -> null
          - verbs             = [
              - "get",
              - "list",
              - "watch",
            ] -> null
        }
    }

  # module.eks.module.eks-init.kubernetes_cluster_role_binding.developer[0] will be created
  + resource "kubernetes_cluster_role_binding" "developer" {
      + id = (known after apply)

      + metadata {
          + generation       = (known after apply)
          + name             = "my:developer"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }

      + role_ref {
          + api_group = "rbac.authorization.k8s.io"
          + kind      = "ClusterRole"
          + name      = "cluster-admin"
        }

      + subject {
          + api_group = "rbac.authorization.k8s.io"
          + kind      = "Group"
          + name      = "my:developer"
          + namespace = "default"
        }
    }

  # module.eks.module.eks-init.kubernetes_cluster_role_binding.developer-prod[0] will be destroyed
  - resource "kubernetes_cluster_role_binding" "developer-prod" {
      - id = "my:developer" -> null

      - metadata {
          - annotations      = {} -> null
          - generation       = 0 -> null
          - labels           = {} -> null
          - name             = "my:developer" -> null
          - resource_version = "841512" -> null
          - uid              = "fa7edb11-bed0-4fa0-b23a-d2e9b031c4a0" -> null
        }

      - role_ref {
          - api_group = "rbac.authorization.k8s.io" -> null
          - kind      = "ClusterRole" -> null
          - name      = "my:developer" -> null
        }

      - subject {
          - api_group = "rbac.authorization.k8s.io" -> null
          - kind      = "Group" -> null
          - name      = "my:developer" -> null
          - namespace = "default" -> null
        }
    }

Plan: 1 to add, 0 to change, 2 to destroy.
module.eks.module.eks-init.kubernetes_cluster_role_binding.developer-prod[0]: Destroying... [id=my:developer]
module.eks.module.eks-init.kubernetes_cluster_role_binding.developer[0]: Creating...
module.eks.module.eks-init.kubernetes_cluster_role_binding.developer-prod[0]: Destruction complete after 1s
module.eks.module.eks-init.kubernetes_cluster_role.developer[0]: Destroying... [id=my:developer]
module.eks.module.eks-init.kubernetes_cluster_role.developer[0]: Destruction complete after 0s
╷
│ Error: clusterrolebindings.rbac.authorization.k8s.io "my:developer" already exists
│
│   with module.eks.module.eks-init.kubernetes_cluster_role_binding.developer[0],
│   on .terraform/modules/eks/eks-init/clusterroles.tf line 55, in resource "kubernetes_cluster_role_binding" "developer":
│   55: resource "kubernetes_cluster_role_binding" "developer" {
│

Panic Output

Steps to Reproduce

1. Set context to prod
2. run terraform apply
3. set context to dev
4. run terraform apply

Expected Behavior

Role binding should be removed before new rolebinding with the same name in metadata.name is created

Actual Behavior

Terraform failed to replace resource

Important Factoids

References

• GH-1234

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[github-actions[bot]]

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.