[terraform-google-workload-identity] recreates resources every time something change

[MoShiKB]

TL;DR

Hey :) I have a problem with the workload identity module, it keeps recreates everytime I add or remove SA from my loop

Expected behavior

update the resources without destroying them first.

Observed behavior

No response

Terraform Configuration

**tfvars:**

project_id = "****"
gsa = [
  {
    gsa_name = "external-secrets-sa"
    project_roles = [
      "****=>roles/secretmanager.secretAccessor",
      "****=>roles/secretmanager.viewer"
    ]
    workload_identity = [
      "external-secrets=>external-secrets"
    ]
  },
  {
    gsa_name = "external-dns-sa"
    project_roles = [
      "****=>roles/dns.admin"
    ]
    workload_identity = [
      "external-dns=>external-dns",
      "external-dns=>external-dns-google",
      "external-dns=>external-dns-cloudflare"
    ]
  }
]


**main.tf:**

locals {
  gsa = {
    for x in var.gsa :
    "${x.gsa_name}" => x
  }

module "service_accounts" {
    source = "[email protected]:*****/*****-terraform-modules.git//gcp/serviceaccount"
    for_each = local.gsa

    project_id = var.project_id
    gsa_name = each.value.gsa_name
    project_roles = each.value.project_roles
    workload_identity = each.value.workload_identity


**module tf:**

locals {
  workload_identity = toset(var.workload_identity)
}

module "service_accounts" {
  source  = "terraform-google-modules/service-accounts/google"
  version = "4.1.0"
  project_id    = var.project_id
  prefix        = var.prefix
  names         = [var.gsa_name]
  project_roles = var.project_roles
  generate_keys = var.generate_keys
}

module "workload_identity" {
  # count = var.workload_identity ? 1 : 0
  for_each = local.workload_identity

  source              = "terraform-google-modules/kubernetes-engine/google//modules/workload-identity"
  project_id          = var.project_id
  annotate_k8s_sa     = false
  use_existing_gcp_sa = true
  use_existing_k8s_sa = true
  name                = var.gsa_name
  # gcp_sa_name         = var.gsa_name
  # namespace           = each.value.ksa_namespace
  # k8s_sa_name         = each.value.ksa_name
  namespace = element(split("=>",each.key),0,)
  k8s_sa_name = element(split("=>",each.key),1,)

  depends_on = [
    module.service_accounts,
  ]
}

Terraform Version

Terraform v1.2.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/external v2.2.2
+ provider registry.terraform.io/hashicorp/google v4.36.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.13.1
+ provider registry.terraform.io/hashicorp/null v3.1.1
+ provider registry.terraform.io/hashicorp/random v3.4.3

Additional information

I have a loop of GSA(you can see at the vars) that uses two modules, service accounts - https://github.com/terraform-google-modules/terraform-google-service-accounts workload-identity from this repo.

it creates the SA using the service account module and then creates workload identity IAM using the workload-identity module.

every time I add or remove a service account from the list, terraform wants to recreate the workload identity IAM.

  # module.service_accounts["external-secrets-sa"].module.workload_identity["external-secrets=>external-secrets"].data.google_service_account.cluster_service_account[0] will be read during apply
  # (depends on a resource or a module with changes pending)
 <= data "google_service_account" "cluster_service_account" {
      + account_id   = "external-secrets-sa"
      + display_name = (known after apply)
      + email        = (known after apply)
      + id           = (known after apply)
      + name         = (known after apply)
      + project      = "*****"
      + unique_id    = (known after apply)
    }

  # module.service_accounts["external-secrets-sa"].module.workload_identity["external-secrets=>external-secrets"].google_service_account_iam_member.main must be replaced
-/+ resource "google_service_account_iam_member" "main" {
      ~ etag               = "BwXooyvL2ZM=" -> (known after apply)
      ~ id                 = "projects/****/serviceAccounts/external-secrets-sa@elementor-hosting-dev.iam.gserviceaccount.com/roles/iam.workloadIdentityUser/serviceAccount:elementor-hosting-dev.svc.id.goog[external-secrets/external-secrets]" -> (known after apply)
      ~ service_account_id = "projects/elementor-hosting-dev/serviceAccounts/external-secrets-sa@elementor-hosting-dev.iam.gserviceaccount.com" -> (known after apply) # forces replacement
        # (2 unchanged attributes hidden)
    }

I tried some suggestions here by using data before the module but it didn't work. Does anyone know if that's a known bug? any workaround I can do to prevent it?

thank you :)

[bharathkkb]

Hi @MoShiKB, I believe what is happening here is the

depends_on = [
    module.service_accounts,
  ]

on WI module causing TF to defer the data source (due to use_existing_gcp_sa) read in that module to apply. This causes resource dependent on info from that data source to be included in the plan as potential change once the data source read is complete. https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/blob/26aa62686db48f774d9d36b36bbb760f41b34f28/modules/workload-identity/main.tf#L80

Does removing that explicit dependency still evaluate correctly?

[MoShiKB]

Hi @MoShiKB, I believe what is happening here is the

depends_on = [
    module.service_accounts,
  ]

on WI module causing TF to defer the data source (due to use_existing_gcp_sa) read in that module to apply. This causes resource dependent on info from that data source to be included in the plan as potential change once the data source read is complete.

https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/blob/26aa62686db48f774d9d36b36bbb760f41b34f28/modules/workload-identity/main.tf#L80

Does removing that explicit dependency still evaluate correctly?

Hey, Thanks for answering. I can't remove the dependency because the workload identity part will run in the same time as the service account creation. even if I

 Error: Invalid template interpolation value
 
   on .terraform/modules/service_accounts.workload_identity/modules/workload-identity/main.tf line 22, in locals:
   22:   gcp_sa_fqn     = "serviceAccount:${local.gcp_sa_email}"
     
      local.gcp_sa_email is null
 
 The expression result is null. Cannot include a null value in a string
 template.

 Error: Invalid template interpolation value
 
   on .terraform/modules/service_accounts.workload_identity/modules/workload-identity/main.tf line 73, in module "annotate-sa":
   73:   kubectl_create_command  = "kubectl annotate --overwrite sa -n ${local.output_k8s_namespace} ${local.k8s_given_name} iam.gke.io/gcp-service-account=${local.gcp_sa_email}"
     
      local.gcp_sa_email is null
 
 The expression result is null. Cannot include a null value in a string
 template.

even if I try to put a dependency: module.service_accounts.service_accounts[0].account_id it gives me an error because of a new service account that doesn't exist. do you know if there is any way to solve it?

[MoShiKB]

Bump

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days