Provider produced inconsistent result after apply

[MarvinKal]

Terraform Version, Provider Version and Kubernetes Version

Terraform v1.0.11
on windows_amd64
+ provider registry.terraform.io/hashicorp/helm v2.4.1
+ provider registry.terraform.io/hashicorp/kubernetes v2.6.1
+ provider registry.terraform.io/ibm-cloud/ibm v1.36.0

Affected Resource(s)

Terraform Configuration Files

provider "kubernetes" {
  host                   = data.ibm_container_cluster_config.cluster_config.host
  token                  = data.ibm_container_cluster_config.cluster_config.token
  cluster_ca_certificate = data.ibm_container_cluster_config.cluster_config.ca_certificate
}

data "ibm_container_cluster_config" "cluster_config" {
  cluster_name_id = ibm_container_vpc_cluster.cluster.id
}

data "ibm_secrets_manager_secret" "jenkins_service_account_secret" {
    instance_id = var.secret_manager_id
    secret_type = "arbitrary"
    secret_id = var.jenkins_service_account_token_secret_id
}

resource "kubernetes_secret" "jenkins_secret" {
  metadata {
    name = "jenkins-service-account-secret"
    namespace = "jenkins-namespace"
    annotations = {
      "kubernetes.io/service-account.name" = "jenkins-service-account" 
    }
  }
  data = {
    "token" = lookup(data.ibm_secrets_manager_secret.jenkins_service_account_secret.secret_data, "payload", "")
  }
  type = "kubernetes.io/service-account-token"
}

resource "kubernetes_service_account" "jenkins_service_account" {
  metadata {
    name      = "jenkins-service-account"
    namespace = "jenkins-namespace"
  }
  secret {
    name = kubernetes_secret.jenkins_secret.metadata.0.name
  }
  automount_service_account_token = false
  depends_on = [kubernetes_secret.jenkins_secret]  
}

Debug Output

https://gist.github.com/MarvinKal/e065d4cc0076c09416f1e52a611c190d

Steps to Reproduce

When I ran "terraform destroy" earlier I wasn't able to delete the kubernetes secret "jenkins-secret", because I wasn't able to connect to the cluster as it has already been destroyed earlier. I got an error message that was something like this when trying to delete the secret:

dial tcp :SomeIpAddresse: connectex: No connection could be made because the target machine actively refused it.

Every other resource was destroyed, except for the secret, which refused to be deleted. So I put a tainted flag on it and was able to delete it and run "terraform apply" again. But now I'm not able to create the secret again and get this error message:

Error: Provider produced inconsistent result after apply

When applying changes to kubernetes_secret.jenkins_secret, provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an unexpected new value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own issue tracker. 

When I'm now trying to remove the tainted flag terraform says the secret doesn't exist, which makes sense, because it hasn't been created again.

Expected Behavior

What should have happened? Terraform should've created the secret without a problem, because the exact same code was working before.

Actual Behavior

Terraform threw an error and the kubernetes secret is not created.

Important Factoids

Maybe this is important, but I'm working within the IBM Cloud.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[MarvinKal]

Okay so I found the solution to my issue here: https://szabo.jp/2021/09/17/terraform-kubernetes_secret-inconsistent-result-after-apply/ . The problem was, that the secret is immediately deleted if the service account it is supposed to be assigned to doesnt exist. I don't really know why that is the case though. So now I just changed the order of creation by making the secret depend on its service account. I don't know if that's the best practice. Here are the changed resources:

resource "kubernetes_service_account" "jenkins_service_account" {
  metadata {
    name      = "jenkins-service-account"
    namespace = "jenkins-namespace"
  }
  secret {
    //name = kubernetes_secret.jenkins_secret.metadata.0.name
    name = "jenkins-service-account-secret"
  }
  automount_service_account_token = false
}

resource "kubernetes_secret" "jenkins_secret" {
  metadata {
    name = "jenkins-service-account-secret"
    namespace = "jenkins-namespace"
    annotations = {
      "kubernetes.io/service-account.name" = "jenkins-service-account"
    }
  }
  data = {
    "token" = lookup(data.ibm_secrets_manager_secret.jenkins_service_account_secret.secret_data, "payload", "")
  }  
  type = "kubernetes.io/service-account-token"
  depends_on = [kubernetes_service_account.jenkins_service_account]  
}

I'd still love me some more insight, especially since in the official documentation it seems as if the service account relies on the secret being created first, which shouldn't work. Official documentation from https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/service_account

resource "kubernetes_service_account" "example" {
  metadata {
    name = "terraform-example"
  }
  secret {
    name = "${kubernetes_secret.example.metadata.0.name}" //reference to secret which means it as to be created first, right?
  }
}

resource "kubernetes_secret" "example" {
  metadata {
    name = "terraform-example"
  }
}

[kyleli666]

I found that the kubernetes_service_account will automatically generate the secret with its name as a prefix, like <sa_name>-token-xxxx.

[github-actions[bot]]

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!