In Terraform Apply for AKS , getting error ""provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an │ unexpected new value: Root resource was present, but now absent."

[Sanghamitra-PERSONAL]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: v1.2.7 on linux_amd64 Kubernetes provider version: v2.13.1 Kubernetes version: 1.24.3

### Affected Resource(s)
Its happening on below resource creations "Terraform Configuration Files" : 
``
resource "kubernetes_secret" "sa" {
  metadata {
    name = "secretName"
    namespace = "namespace"
    annotations = {
      "kubernetes.io/service-account.name" = "serviceAccountName"
       }
  }
  type = "kubernetes.io/service-account-token"
}

resource "kubernetes_service_account" "sa" {
  metadata {
    name = "serviceAccountName"
    namespace = "namespace"
  }
  secret {
    name = kubernetes_secret.sa.metadata.0.name
  }
}

Terraform Apply's Output :

module.kubernetes_service_account["01"].kubernetes_secret.sa: Creating...
╷
│ Error: Provider produced inconsistent result after apply
│ 
│ When applying changes to
│ module.kubernetes_service_account["01"].kubernetes_secret.sa, provider
│ "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced an
│ unexpected new value: Root resource was present, but now absent.
│ 
│ This is a bug in the provider, which should be reported in the provider's
│ own issue tracker.

Expected Behavior

This error should not have come. The above resources would have got created without any errors. This behaviour I am seeing only this week.

Actual Behavior

This error was not there before this week.

Steps to Reproduce

terraform init
terraform plan -var-file ..\cluster.tfvars
terraform apply -var-file ..\cluster.tfvars

References

I think I am first to report it.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[Sanghamitra-PERSONAL]

Today I have tested the same code in below ways in K8s 1.23.5 AKS : a. With K8s version 1.23.5 and k8s provider version 2.13.1 or 2.12.1 , its failing on secret creation with the same msg : module.kubernetes_service_account["01"].kubernetes_secret.sa: Creating... b. With K8s version 1.23.5 and k8s provider version 2.12.1 or 2.13.1 , I used below and its successful:

data "kubernetes_secret" "sa" {
  metadata {
    name = kubernetes_service_account.sa.default_secret_name
    namespace = "namespace"
  }
}

resource "kubernetes_service_account" "sa" {
  metadata {
    name = "serviceAccountName"
    namespace = "namespace"
  }
}

Conclusion :

• I see problem in resource "kubernetes_secret" creation and everytime it was failing "module.kubernetes_service_account["01"].kubernetes_secret.sa: Creating..."

• Hence we cannot go to K8s version 1.24 since in k8s version 1.24 secret wont be created automatically by service account . We have to use resource "kubernetes_secret" which is failing clearly.

[Sanghamitra-PERSONAL]

Hi Team, Please have a note on this issue.

Thanks, Sanghamitra

[jrhouston]

I'm not able to reproduce this with just the config you've supplied – I can see from your output that you are perhaps using for_each with the resource? Could you share more of the config?

[Sanghamitra-PERSONAL]

Hi @jrhouston , Please see the complete code below :

• resourceDeclaration :

resource "kubernetes_secret" "sa" {
        metadata {
            name = local.serviceAccount.config.secretName
            namespace = local.serviceAccount.config.namespace
            annotations = {
                "kubernetes.io/service-account.name" = "${local.serviceAccount.config.serviceAccountName}"
             }
        }
       type = "kubernetes.io/service-account-token"
    }

resource "kubernetes_service_account" "sa" {
       metadata {
       name = local.serviceAccount.config.serviceAccountName
       namespace = local.serviceAccount.config.namespace
      }
       secret {
          name = "${kubernetes_secret.sa.metadata.0.name}"
      }
     }

     output "token" {
          value = kubernetes_secret.sa.data.token
         }

=======================================

• We have called the above here in main.tf :

  module "service_account" { source = "../../" for_each = { for s in local.boundary.xyz : s.serviceConnectionID => s } serviceAccount = merge( { config = each.value
  } ) depends_on = [ module.namespaces ] }

====================================

• tfvar file which has the values :

xyz= [ { serviceConnectionID = "01" secretName = "secretName" serviceAccountName = "serviceAccountName" namespace = "namespace" authorizationType = "ServiceAccount" } ]

Thanks, Sanghamitra

[arybolovlev]

Hi @Sanghamitra-PERSONAL,

I think that issue should be addressed by this PR. The problem here is that when you create a service account token via kubernetes_secret it doesn't wait for the token to be created and as a result of the first run, you get an empty data map response. If you run your code in standalone mode, you will get an error similar to:

│ Error: Missing map element
│
│   on main.tf line 21, in output "token":
│   21:   value = kubernetes_secret.sa.data.token
│     ├────────────────
│     │ kubernetes_secret.sa.data has a sensitive value
│
│ This map does not have an element with the key "token".

But the second run should be successful. Please keep an eye on the release notes and try it out once the fix is available.

Thanks.

[Sanghamitra-PERSONAL]

Hi @arybolovlev ,

Thanks for your attention on this issue. I can see the PR is merged yesterday , but there is no new release created for the new main. Could you please let me know the release number I need to pull the provider ?

Thanks, Sanghamitra

[github-actions[bot]]

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!