Cannot apply changes to `kubernetes_manifest` directly after it has been imported

[theKlisha]

Hello,

Terraform fails to run apply after kubernetes_manifest that is defining kubernetes deployment has been imported if doing so would modify order of environment variables.

Terraform Version, Provider Version and Kubernetes Version

sh-3.2$ terraform -v
Terraform v1.10.1
on darwin_arm64
+ provider registry.terraform.io/hashicorp/kubernetes v2.34.0

sh-3.2$ kubectl version
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.31.0

Affected Resource(s)

• kubernetes_manifest

Terraform Configuration Files

terraform {
  backend "local" {
    path = "terraform.tfstate"
  }
}

terraform {
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">=2.34.0"
    }
  }
  required_version = ">= 1.8.0"
}

provider "kubernetes" {
  config_path    = "~/.kube/config"
  config_context = "minikube"
}

resource "kubernetes_manifest" "manifest" {
  manifest = {
    apiVersion = "apps/v1"
    kind       = "Deployment"

    metadata = {
      name      = "test"
      namespace = "default"
    }

    spec = {
      replicas = 1

      selector = {
        matchLabels = { app = "test" }
      }

      template = {
        metadata = {
          labels = { app = "test" }
        }

        spec = {
          containers = [
            {
              name  = "nginx"
              image = "nginx"
              env = [
                { name = "A", valueFrom = { fieldRef = { apiVersion = "v1", fieldPath = "metadata.uid" } } },
                { name = "B", value = "B" },
              ]
            }
          ]
        }
      }
    }
  }
}

Debug Output

https://gist.github.com/theKlisha/4ef98730ba5836cc4fe7d1168b39e930

Steps to Reproduce

1. terraform apply
2. terraform state rm kubernetes_manifest.manifest
3. terraform import kubernetes_manifest.manifest "apiVersion=apps/v1,kind=Deployment,namespace=default,name=test"
4. modify terraform configuration: change order of env variables

          containers = [
            {
              name  = "nginx"
              image = "nginx"
              env = [
-               { name = "A", valueFrom = { fieldRef = { apiVersion = "v1", fieldPath = "metadata.uid" } } },
                { name = "B", value = "B" },
+               { name = "A", valueFrom = { fieldRef = { apiVersion = "v1", fieldPath = "metadata.uid" } } },
              ]
            }
          ]

5. terraform apply

Expected Behavior

apply should succeed

Actual Behavior

apply fails with:

Error: API response status: Failure

  with kubernetes_manifest.manifest,
  on main.tf line 22, in resource "kubernetes_manifest" "manifest":
  22: resource "kubernetes_manifest" "manifest" {

Deployment.apps "test" is invalid:
[spec.template.spec.containers[0].env[0].valueFrom: Invalid value: "": may
not be specified when `value` is not empty,
spec.template.spec.containers[0].env[1].valueFrom: Invalid value: "": may not
be specified when `value` is not empty]

Error: Kubernetes API Error: Invalid Deployment.apps [test]

  with kubernetes_manifest.manifest,
  on main.tf line 22, in resource "kubernetes_manifest" "manifest":
  22: resource "kubernetes_manifest" "manifest" {


Error: spec.template.spec.containers[0].env[0].valueFrom

  with kubernetes_manifest.manifest,
  on main.tf line 22, in resource "kubernetes_manifest" "manifest":
  22: resource "kubernetes_manifest" "manifest" {

Invalid value: "": may not be specified when `value` is not empty

Error: spec.template.spec.containers[0].env[1].valueFrom

  with kubernetes_manifest.manifest,
  on main.tf line 22, in resource "kubernetes_manifest" "manifest":
  22: resource "kubernetes_manifest" "manifest" {

Invalid value: "": may not be specified when `value` is not empty

Important Factoids

1. Wen you do one additional apply before changing the order everything goes fine.
2. One workaround we found was to modify the sate manually by copying value of resources[0].instances[0].attributes.object to resources[0].instances[0].attributes.manifest after import (indices for the provided example). This causes significant amount of drift but only on terraform state side, whats more important, apply goes smoothly.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[tavlima]

I'm facing a similar error when trying to add a new env item to the top of the list. Probably the same root cause as #1980.

[MathieuDR]

I'm facing a similar error when trying to add a new env item to the top of the list. Probably the same root cause as #1980.

Same here. Just trying to add an env variable :')