terraform-provider-kubernetes icon indicating copy to clipboard operation
terraform-provider-kubernetes copied to clipboard

Published 20 hours ago • verified publisher icon hashicorp

kubernetes_manifest resources panics when a value depends on other resources

Open junaid-ali opened this issue 3 years ago • 11 comments

I'm creating a configmap where one of the fields in the data section depends on another value. Running terraform apply causes a panic

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 0.14.11
Kubernetes provider version: 1.28
Kubernetes version: 1.20

Affected Resource(s)

kubernetes_manifest

Terraform Configuration Files

resource "kubernetes_manifest" "aws-auth" {
  manifest = {
    apiVersion = "v1"
    kind       = "ConfigMap"
    data       = {
      mapAccounts = yamlencode([])
      mapRoles    = yamldecode(module.eks.aws_auth_configmap_yaml)["data"].mapRoles <--- this value depends
      mapUsers    = yamlencode(local.map_users)
    }
    metadata = {
      name      = "aws-auth"
      namespace = "kube-system"
      labels = {
        "app.kubernetes.io/managed-by" = "Terraform"
      }
    }
  }

  depends_on = [module.eks.self_managed_node_groups]
}

Debug Output

Panic Output

panic: ElementKeyString("mapRoles"): can't use tftypes.DynamicPseudoType as tftypes.String

Steps to Reproduce

References

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

junaid-ali avatar Feb 18 '22 06:02 junaid-ali

Hi @junaid-ali I think you have reported this issue on the wrong project. Your example uses the kubectl_manifest resource which is part of the kubectl provider. The resource supported by this project is kubernetes_manifest.

jrhouston avatar Feb 23 '22 03:02 jrhouston

@jrhouston, thanks for pointing out the mistake. I was testing the kubectl_manifest as well from kubectl provider, and mistakenly pasted that manifest instead of the kubernetes_manifest. I have updated the description since the issue did appear in kubernetes provider's kuberentes_manifest resource.

junaid-ali avatar Feb 23 '22 07:02 junaid-ali

What type do you expect yamldecode(module.eks.aws_auth_configmap_yaml)["data"].mapRoles to be?

It looks like terraform is not able to determine the type of it, but the value expeected should be of type string. Depending on which type you're getting there, it needs to be converted to a string.

alexsomesan avatar Mar 02 '22 16:03 alexsomesan

mapRoles contains a list of AWS IAM role ARNs + other attributes and passed to the configmap. When a new role is to be added to the list, that is yet to be created by terraform, the above error occurs.

P.S: module.eks.aws_auth_configmap_yaml is a rendered configmap, and the data looks like below:

data:
  mapRoles: |
    - rolearn: arn:aws:iam::111122223333:role/eksctl-my-cluster-nodegroup-standard-wo-NodeInstanceRole-1WP3NUE3O6UCF
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: arn:aws:iam::111122223333:role/eksctl-my-cluster-nodegroup-standard-another-one
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: <A new role that's yet to be created by module.eks>
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes

I yaml decode the rendered configmap to add additional AWS IAM users, or roles, or accounts.

junaid-ali avatar Mar 02 '22 16:03 junaid-ali

Hello, I do have the same issue when trying to get value from a generated output from a module

Terraform code

resource "kubernetes_manifest" "service_account" {
  manifest = {
    apiVersion = "v1"
    kind       = "ServiceAccount"
    metadata = {
      name      = local.service_account_name
      namespace = local.namespace
      annotations = {
        "eks.amazonaws.com/role-arn" = module.iam_secrets_role.iam_role_arn
      }
    }
  }
}

Plan output

│ Error: Plugin did not respond
│
│   with kubernetes_manifest.service_account,
│   on 05_main.tf line 43, in resource "kubernetes_manifest" "service_account":
│   43: resource "kubernetes_manifest" "service_account" {
│
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).PlanResourceChange call. The plugin logs may contain
│ more details.
╵

Stack trace from the terraform-provider-kubernetes_v2.8.0_x5 plugin:

panic: ElementKeyString("eks.amazonaws.com/role-arn"): can't use tftypes.DynamicPseudoType as tftypes.String

goroutine 130 [running]:
github.com/hashicorp/terraform-plugin-go/tftypes.NewValue(...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/vendor/github.com/hashicorp/terraform-plugin-go/tftypes/value.go:277
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.morphObjectToType(0x23e9a48, 0xc000c6d920, 0x1e61c80, 0xc000c6cd20, 0x23e99f8, 0xc00229cde0, 0xc000d901e0, 0x2372a10, 0x203000, 0x203000, ...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:342 +0x16a6
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.ValueToType(0x23e9a48, 0xc000c6d920, 0x1e61c80, 0xc000c6cd20, 0x23e99f8, 0xc00229cde0, 0xc000d901e0, 0x2, 0x10, 0x0, ...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:36 +0x709
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.morphObjectToType(0x23e9a48, 0xc000c77110, 0x1e61c80, 0xc000342210, 0x23e9a48, 0xc0022bf560, 0xc000d901c8, 0x23e9b38, 0xc0012a1e00, 0x1daf560, ...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:314 +0x425
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.ValueToType(0x23e9a48, 0xc000c77110, 0x1e61c80, 0xc000342210, 0x23e9a48, 0xc0022bf560, 0xc000d901c8, 0x1, 0xc0012a1e00, 0x1daf560, ...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:36 +0x709
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.morphObjectToType(0x23e9a48, 0xc000c7e090, 0x1e61c80, 0xc0012a1c20, 0x23e9a48, 0xc0022bf590, 0xc000d90180, 0x0, 0x0, 0x0, ...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:314 +0x425
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.ValueToType(0x23e9a48, 0xc000c7e090, 0x1e61c80, 0xc0012a1c20, 0x23e9a48, 0xc0022bf590, 0xc000d90180, 0x2, 0xe, 0x0, ...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:36 +0x709
github.com/hashicorp/terraform-provider-kubernetes/manifest/provider.(*RawProviderServer).PlanResourceChange(0xc00062e200, 0x23d70d8, 0xc0010ac880, 0xc000384960, 0xc00035d8e8, 0xc001c11b01, 0x1)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/manifest/provider/plan.go:315 +0x1cbb
github.com/hashicorp/terraform-plugin-mux.SchemaServer.PlanResourceChange(0xc0012a0480, 0xc0012a04b0, 0xc0010d4ca0, 0x2, 0x2, 0xc00062e180, 0x1, 0x23d70d8, 0xc0010ac880, 0xc000384960, ...)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/vendor/github.com/hashicorp/terraform-plugin-mux/schema_server.go:276 +0x98
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).PlanResourceChange(0xc000140000, 0x23d7180, 0xc0010ae1b0, 0xc00193b180, 0x0, 0x0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server/server.go:777 +0x7f0
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_PlanResourceChange_Handler(0x201dc80, 0xc000140000, 0x23d7180, 0xc0010ae1b0, 0xc0019442a0, 0x0, 0x23d7180, 0xc0010ae1b0, 0xc000056a80, 0x31f)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/vendor/github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:367 +0x214
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001fe380, 0x23eda78, 0xc000ea41a0, 0xc0011d67e0, 0xc0012a0570, 0x318f8c8, 0x0, 0x0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/vendor/google.golang.org/grpc/server.go:1282 +0x52b
google.golang.org/grpc.(*Server).handleStream(0xc0001fe380, 0x23eda78, 0xc000ea41a0, 0xc0011d67e0, 0x0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/vendor/google.golang.org/grpc/server.go:1616 +0xd0c
google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0001169c0, 0xc0001fe380, 0x23eda78, 0xc000ea41a0, 0xc0011d67e0)
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/vendor/google.golang.org/grpc/server.go:921 +0xab
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /opt/teamcity-agent/work/5d79fe75d4460a2f/src/github.com/hashicorp/terraform-provider-kubernetes/vendor/google.golang.org/grpc/server.go:919 +0x1fd

Error: The terraform-provider-kubernetes_v2.8.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

To solve that I have to comment out the annotation, apply so that the module can be created then uncomment the annotation that depends on the generated module

WillerWasTaken avatar Mar 09 '22 09:03 WillerWasTaken

@WillerWasTaken thanks for the added context.

It seems the provider isn't correctly handling some unknown values at plan time. I'm looking into it.

alexsomesan avatar Mar 09 '22 10:03 alexsomesan

I encountered this bug too when dealing with unknown values in the manifest at plan time.

 can't use tftypes.DynamicPseudoType as tftypes.String

My workaround was to wrap the kubernetes_manifest resource in a module, and explicitly cast the unknown variable to a string.

Pseudo code

variable "security_group_id" {
  description = "ID of Security Group."
  type        = string
}

resource "kubernetes_manifest" "security_group" {
  manifest = {
    apiVersion = "vpcresources.k8s.aws/v1beta1"
    kind       = "SecurityGroupPolicy"
    metadata = {
      name      = "foo"
      namespace = "bar"
    }
    spec = {
      podSelector = {
        matchLabels = {
          role = "my-role-label"
        }
      }
      securityGroups = {
        groupIds = [var.security_group_id]
      }
    }
  }
}

Now that i think of it tostring() might have worked as well..

Hope this helps somebody!

sjoerd-dijkstra avatar May 20 '22 11:05 sjoerd-dijkstra

@sjoerd-dijkstra which provider version are you using?

Also, can you also share the module block that you use to invoke that module?

alexsomesan avatar May 20 '22 11:05 alexsomesan

@sjoerd-dijkstra which provider version are you using?

Also, can you also share the module block that you use to invoke that module?

Sure! So i'm using the kubernetes_manifest resource to create custom resources that are not supported by the provider, to use in our EKS cluster specifically. The manifest is conditionally created, and uses variables that are only known during/after running teraform apply.

k8s provider specs

provider "registry.terraform.io/hashicorp/kubernetes" {
  version     = "2.11.0"
  constraints = "~> 2.0"
  hashes = [
    "h1:T65SZhN/tQgsAsHe/G5PCgpjofi+aTKPZ+nZg6WOJpc=",
    "h1:pJiAJwZKUaoAJ4x+3ONJkwEVkjrwGROCGFgj7noPO58=",
    "zh:143a19dd0ea3b07fc5e3d9231f3c2d01f92894385c98a67327de74c76c715843",
    "zh:1fc757d209e09c3cf7848e4274daa32408c07743698fbed10ee52a4a479b62b6",
    "zh:22dfebd0685749c51a8f765d51a1090a259778960ac1cd4f32021a325b2b9b72",
    "zh:3039b3b76e870cd8fc404cf75a29c66b171c6ba9b6182e131b6ae2ca648ec7c0",
    "zh:3af0a15562fcab4b5684b18802e0239371b2b8ff9197ed069ff4827f795a002b",
    "zh:50aaf20336d1296a73315adb66f7687f75bd5c6b1f93a894b95c75cc142810ec",
    "zh:682064fabff895ec351860b4fe0321290bbbb17c2a410b62c9bea0039400650e",
    "zh:70ac914d5830b3371a2679d8f77cc20c419a6e12925145afae6c977c8eb90934",
    "zh:710aa02cccf7b0f3fb50880d6d2a7a8b8c9435248666616844ba71f74648cddc",
    "zh:88e418118cd5afbdec4984944c7ab36950bf48e8d3e09e090232e55eecfb470b",
    "zh:9cef159377bf23fa331f8724fdc6ce27ad39a217a4bae6df3b1ca408fc643da6",
    "zh:f569b65999264a9416862bca5cd2a6177d94ccb0424f3a4ef424428912b9cb3c",
  ]
}

module block

Simplified:

module "k8s_security_group" {
  count = var.security_group_enabled ? 1 : 0

  source = "./modules/k8s-security-group"

  security_group_id = module.security_group_pod[count.index].security_group_id # this value is not known during plan
}

Now the content of the module is pretty much the piece of code i posted above, with only a few more variables.

Tbh, i think the tostring() terraform function could have done the trick as well. Not entirely sure though, as I haven't tested that.

Is this the info you need? Hope it helps!

sjoerd-dijkstra avatar May 20 '22 12:05 sjoerd-dijkstra

That's all good info, thanks! Which error are you seeing when not using tostring() ?

alexsomesan avatar May 20 '22 12:05 alexsomesan 


│ Error: Request cancelled
| ..... <ommited>
│ 
│ The plugin.(*GRPCProvider).UpgradeResourceState request was cancelled.
╵
Releasing state lock. This may take a few moments...
Stack trace from the terraform-provider-kubernetes_v2.11.0_x5 plugin:
panic: ElementKeyInt(0): can't use tftypes.DynamicPseudoType as tftypes.String
goroutine 142 [running]:
github.com/hashicorp/terraform-plugin-go/tftypes.NewValue(...)
	github.com/hashicorp/[email protected]/tftypes/value.go:277
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.morphTupleIntoType({{0x2083708?, 0xc000fa0630?}, {0x1a0be40?, 0xc0016d28a0?}}, {0x20835c8?, 0xc0023ddbf0?}, 0xc0026c0ed0)
	github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:201 +0xb2b
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.ValueToType({{0x2083708?, 0xc000fa0630?}, {0x1a0be40?, 0xc0016d28a0?}}, {0x20835c8, 0xc0023ddbf0}, 0x0?)
	github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:30 +0x5f5
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.morphObjectToType({{0x2083668?, 0xc000fa0ab0?}, {0x1b0e860?, 0xc000fa02a0?}}, {0x2083668?, 0xc0023ddc20?}, 0xc0026c0eb8)
	github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:314 +0xb85
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.ValueToType({{0x2083668?, 0xc000fa0ab0?}, {0x1b0e860?, 0xc000fa02a0?}}, {0x2083668, 0xc0023ddc20}, 0x1c94060?)
	github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:36 +0x4a5
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.morphObjectToType({{0x2083668?, 0xc000fa0ea0?}, {0x1b0e860?, 0xc000f964e0?}}, {0x2083668?, 0xc00282c510?}, 0xc0026c0e40)
	github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:314 +0xb85
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.ValueToType({{0x2083668?, 0xc000fa0ea0?}, {0x1b0e860?, 0xc000f964e0?}}, {0x2083668, 0xc00282c510}, 0x1891635?)
	github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:36 +0x4a5
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.morphObjectToType({{0x2083668?, 0xc000fa1110?}, {0x1b0e860?, 0xc000f83bf0?}}, {0x2083668?, 0xc003152f90?}, 0xc0026c0dc8)
	github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:314 +0xb85
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.ValueToType({{0x2083668?, 0xc000fa1110?}, {0x1b0e860?, 0xc000f83bf0?}}, {0x2083668, 0xc003152f90}, 0xc001878d47?)
	github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/morph.go:36 +0x4a5
github.com/hashicorp/terraform-provider-kubernetes/manifest/provider.(*RawProviderServer).PlanResourceChange(0xc00122ca00, {0x2080160, 0xc001281020}, 0xc001f020a0)
	github.com/hashicorp/terraform-provider-kubernetes/manifest/provider/plan.go:327 +0x313b
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.PlanResourceChange({0xc000de89c0, 0xc000de8a20, {0xc0001543a0, 0x2, 0x2}, 0xc000de89f0, 0xc000db0d70, 0xc000abfcc0, 0xc000de8a50}, {0x20800b8, ...}, ...)
	github.com/hashicorp/[email protected]/tf5muxserver/mux_server_PlanResourceChange.go:27 +0x142
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).PlanResourceChange(0xc0003ead20, {0x2080160?, 0xc0012804b0?}, 0xc000358000)
	github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:779 +0x515
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_PlanResourceChange_Handler({0x1cb7000?, 0xc0003ead20}, {0x2080160, 0xc0012804b0}, 0xc0018ce060, 0x0)
	github.com/hashicorp/[email protected]/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:367 +0x170
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000228700, {0x2084158, 0xc000a86680}, 0xc001846000, 0xc000e44720, 0x2e6df68, 0x0)
	google.golang.org/[email protected]/server.go:1282 +0xccf
google.golang.org/grpc.(*Server).handleStream(0xc000228700, {0x2084158, 0xc000a86680}, 0xc001846000, 0x0)
	google.golang.org/[email protected]/server.go:1619 +0xa1b
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/[email protected]/server.go:921 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/[email protected]/server.go:919 +0x28a
Error: The terraform-provider-kubernetes_v2.11.0_x5 plugin crashed!
This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

sjoerd-dijkstra avatar May 20 '22 12:05 sjoerd-dijkstra

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

github-actions[bot] avatar May 21 '23 00:05 github-actions[bot]

Can confirm that had the same issue while trying to install Crossplane's DeploymentRuntimeConfig kuberenetes_manifest and I had to change this:

"spec" = {
      "serviceAccountTemplate" = {
        "metadata" = {
          "annotations" = {
            "eks.amazonaws.com/role-arn" = module.iam_role_oidc_kubernetes_crossplane.iam_role_arn
          }
        }
      }
    }

To this:

"spec" = {
      "serviceAccountTemplate" = {
        "metadata" = {
          "annotations" = {
            "eks.amazonaws.com/role-arn" = tostring(module.iam_role_oidc_kubernetes_crossplane.iam_role_arn)
          }
        }
      }
    }

denzhel avatar Jan 16 '24 21:01 denzhel