volume_mount is recreated every apply

[ronendayan]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 1.2.6
Kubernetes provider version: 2.13.1
Kubernetes version: 1.22.11-gke.400

Terraform Configuration Files

resource "kubernetes_service_account" "service_account" {
  automount_service_account_token = var.automount_service_account_token
  metadata {
    name        = var.name
    namespace   = var.namespace
    annotations = var.annotations
  }
}

resource "kubernetes_deployment" "deployment" {
...
   volume_mount {
       mount_path = "/var/run/secrets/kubernetes.io/serviceaccount"
       name = kubernetes_service_account.service_account.default_secret_name
    }
  volume {
    secret {
      secret_name = kubernetes_service_account.service_account.default_secret_name
      }
    name = kubernetes_service_account.service_account.default_secret_name
  }
}

Output

  # module.create_kube_config.module.prometheus_deployment.kubernetes_deployment.deployment will be updated in-place
  ~ resource "kubernetes_deployment" "deployment" {
        id               = "monitoring/monitoring-prometheus"
        # (1 unchanged attribute hidden)

      ~ spec {
            # (5 unchanged attributes hidden)

          ~ template {

              ~ spec {
                    # (12 unchanged attributes hidden)

                  ~ container {
                        name                       = "monitoring-prometheus"
                        # (9 unchanged attributes hidden)

                      + volume_mount {
                          + mount_path        = "/var/run/secrets/kubernetes.io/serviceaccount"
                          + mount_propagation = "None"
                          + name              = "prometheus-token-8dqxg"
                          + read_only         = false
                        }

                        # (3 unchanged blocks hidden)
                    }

                  + volume {
                      + name = "prometheus-token-8dqxg"

                      + secret {
                          + default_mode = "0644"
                          + secret_name  = "prometheus-token-8dqxg"
                        }
                    }

                    # (1 unchanged block hidden)
                }

                # (1 unchanged block hidden)
            }

            # (2 unchanged blocks hidden)
        }

        # (2 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Expected Behavior

Update the state file with the volume_mount and volume

Actual Behavior

kubernetes_service_account under GKE with workload_identity generate it own kube secret prometheus-token-8dqxg And each run it want to add it, but it not updating the state file

Important Factoids

Running on GKE with workload_identity enabled

[arybolovlev]

Hi @ronendayan,

I was trying to reproduce this issue, but cannot. Here is the TF code I use:

resource "kubernetes_service_account" "this" {
  automount_service_account_token = true
  metadata {
    name = "this"
  }
}

resource "kubernetes_deployment_v1" "this" {
  metadata {
    name = "this"
    labels = {
      this = "this"
    }
  }

  spec {
    replicas = 2

    selector {
      match_labels = {
        this = "this"
      }
    }

    template {
      metadata {
        labels = {
          this = "this"
        }
      }

      spec {
        container {
          image = "nginx:1.21.6"
          name  = "this"

          volume_mount {
            mount_path = "/var/run/secrets/kubernetes.io/serviceaccount"
            name       = kubernetes_service_account.this.default_secret_name
          }
        }

        volume {
          secret {
            secret_name = kubernetes_service_account.this.default_secret_name
          }
          name = kubernetes_service_account.this.default_secret_name
        }
      }
    }
  }
}

First apply -- no issues, second apply -- no changes. In both cases, I am getting a deprecation message, but this is expected.

In the terraform state file I can see no issues:

    "volume_mount": [
      {
        "mount_path": "/var/run/secrets/kubernetes.io/serviceaccount",
        "mount_propagation": "None",
        "name": "this-token-zk9jg",
        "read_only": false,
        "sub_path": ""
      }
    ],
    "working_dir": ""
  }
],
"volume": [
  {
    "secret": [
      {
        "default_mode": "0644",
        "items": [],
        "optional": false,
        "secret_name": "this-token-zk9jg"
      }
    ],
  }

One of the pods:

Containers:
  this:
    Image:          nginx:1.21.6
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from this-token-zk9jg (rw)
Volumes:
  this-token-zk9jg:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  this-token-zk9jg
    Optional:    false

Kubernetes cluster: v1.22.12-gke.2300, Workload Identity is Enabled. TF Provider: v2.13.1

Please let me know if I have missed something in my reproduction.

Thanks.

[ronendayan]

I tested it again and found that if you add: service_account_name = kubernetes_service_account.this.metadata[0].name It will be reproduced

Also weird stuff, once I add service_account_name, can't remove it

[arybolovlev]

Thank you, @ronendayan.

I could reproduce it and I can confirm that this is expected behavior. Please find my explanation below.

• Attribute service_account_name configures the service account for a pod as it is described here. It means, that the corresponding volume and volumeMount will be created automatically. So you can, but don't have to specify them in your manifest:

  template:
    spec:
      automountServiceAccountToken: true
        # Created automatically since I have specified service account
        volumeMounts:
        - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          mountPropagation: None
          name: this-token-27kwk
      serviceAccount: this
      # This I have configured
      serviceAccountName: this
      # Created automatically since I have specified service account
      volumes:
      - name: this-token-27kwk
        secret:
          defaultMode: 420
          optional: false
          secretName: this-token-27kwk

• In our provider, we intentionally ignore volume and volumeMount that were created by Kubernetes due to configured service account. If we don't ignore it, then everyone will have to configure these 3 attributes together(serviceAccountName, volume, volumeMount) which doesn't match with the Kubernetes logic. Why so? The Kubernetes API returns all 3 attributes once you have configured just one of them serviceAccountName. Then Terraform have to save this to the state file, but next plan will show difference between the state file and the actual code. And since volume and volumeMount are only in the state file, Terraform will try to remove them. That is why they are ignored. By 'ignored' I mean they never end up in the state file.

I believe, that in your case, you can simply switch to the attribute service_account_name and rid of volume.secret and volume_mount.

I hope that helps.

Thank you.

[ronendayan]

Thanks, set it up and it works fine

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.