Error: Provider produced inconsistent result after apply - CSI Driver on AKS

[alessandrogiorgianni]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 1.2.2
Kubernetes provider version: 2.11.0
Kubernetes version: 1.22.6

Affected Resource(s)

• kubernetes_manifest

Terraform Configuration Files

resource "kubernetes_manifest" "csi-pod" {
  manifest = {
    apiVersion = "v1"
    kind       = "Pod"
    metadata = {
      name      = "busybox-secrets-store-inline-user-msi"
      namespace = "default"
    }
    spec = {
      containers = [
        {
          command = [
            "/bin/sleep",
            "10000"
          ]
          image = "k8s.gcr.io/e2e-test-images/busybox:1.29-1"
          name  = "busybox"
          volumeMounts = [
            {
              mountPath = "/mnt/secrets-store"
              name      = "secrets-store01-inline"
              readOnly  = true
            }
          ]
        }
      ]
      volumes = [
        {
          csi = {
            driver   = "secrets-store.csi.k8s.io"
            readOnly = true
            volumeAttributes = {
              secretProviderClass = "azure-kvname-user-msi"
            }
          }
          name = "secrets-store01-inline"
        }
      ]
    }
  }
}

Debug Output

Panic Output

Steps to Reproduce

terraform apply

Expected Behavior

The equivalent kubectl apply succeeds.

Actual Behavior

Error: Provider produced inconsistent result after apply
When applying changes to kubernetes_manifest.csi-pod, provider "provider[\"registry.terraform.io/hashicorp/kubernetes\"]" produced 
an  unexpected new value: .object: wrong final value type: attribute "spec": attribute "volumes": tuple required.
This is a bug in the provider, which should be reported in the provider's own issue tracker.

Important Factoids

I am installing the CSI driver for AKS in Azure. The pod gets created and the CSI driver works properly. However TF detects some drift in the returned object and taints the manifest. I tried to use computed_fields but couldn't get the wanted result.

References

• GH-1234

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[chrisbalmer]

Same error on vanilla kubernetes with longhorn storage provider.

When applying changes to kubernetes_manifest.pod_jira, provider "provider["registry.terraform.io/hashicorp/kubernetes"]" produced an unexpected new value: .object: wrong final value type: attribute "spec": attribute "volumes": tuple required.

[chrisbalmer]

Swapping from a kubernetes_manifest resource to a kubernetes_pod_v1 resource fixed the issue for me.

[github-actions[bot]]

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

[scorsi]

Swapping from a kubernetes_manifest resource to a kubernetes_pod_v1 resource fixed the issue for me.

Yes it works, thanks !

resource "kubernetes_pod" "busybox" {
    metadata {
        name = "busybox-secrets-store-inline"
        namespace = local.namespace
    }

    spec {
      container {
        name = "busybox"
        image = "registry.k8s.io/e2e-test-images/busybox:1.29-1"
        command = ["/bin/sleep", "10000"]
        volume_mount {
          name       = "secrets-store-inline"
          mount_path = "/mnt/secrets-store"
          read_only  = true
        }
      }

      volume {
        name = "secrets-store-inline"
        csi {
          driver = "secrets-store.csi.k8s.io"
          read_only = true
          volume_attributes = {
            secretProviderClass = "azure-sync"
          }
        }
      }
    }
}