The "count" value depends on resource attributes that cannot be determined until apply

[eugeneromero]

Hello!

I have the following Terraform code inside of a module:

data "kubectl_path_documents" "esCustomResourcesManifests" {
  pattern = "modules/elasticsearch/all-in-one-1.3.0.yaml"
  disable_template = true
}

resource "kubectl_manifest" "esCustomResources" {
  count     = length(data.kubectl_path_documents.esCustomResourcesManifests.documents)
  yaml_body = element(data.kubectl_path_documents.esCustomResourcesManifests.documents, count.index)
}

When running terraform plan, I get:

Error: Invalid count argument

  on modules/elasticsearch/elasticSearch.tf line 24, in resource "kubectl_manifest" "esCustomResources":
  24:   count     = length(data.kubectl_path_documents.esCustomResourcesManifests.documents)

The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.

This does not seem to be the same as issue #58, since I am not pulling any external variables. Is there something obvious I am missing?

Versions: Terraform v0.13.5 provider registry.terraform.io/gavinbunney/kubectl v1.9.1

YAML file: https://download.elastic.co/downloads/eck/1.3.0/all-in-one.yaml

Thanks in advance for your help!

[sathish-kumar-narayanan]

Any updates on this issue ? Am facing similar problem, this happens only if it is used inside a module

[gavinbunney]

Hi @eugeneromero my guess is the pathing isn't quite right as you need to have the ${path.module}/ as a prefix for the pattern path (obviously fixing up the relative path to that file)... so try something like this:

data "kubectl_path_documents" "esCustomResourcesManifests" {
  pattern = "${path.module}/modules/elasticsearch/all-in-one-1.3.0.yaml"
  disable_template = true
}

resource "kubectl_manifest" "esCustomResources" {
  count     = length(data.kubectl_path_documents.esCustomResourcesManifests.documents)
  yaml_body = element(data.kubectl_path_documents.esCustomResourcesManifests.documents, count.index)
}

[sunsingerus]

Hi @gavinbunney hit the same issue:

data "kubectl_path_documents" "dashboard-manifests" {
  pattern = "${path.module}/yaml/dashboard.yaml"
}

resource "kubectl_manifest" "dashboard" {
  count     = length(data.kubectl_path_documents.dashboard-manifests.documents)
  yaml_body = element(data.kubectl_path_documents.dashboard-manifests.documents, count.index)
}

Error in resource "kubectl_manifest" "dashboard":

   6:   count     = length(data.kubectl_path_documents.dashboard-manifests.documents)

The "count" value depends on resource attributes that cannot be determined

[sunsingerus]

kubectl provider is the latest one, 1.10.0

[sunsingerus]

this happens only if it is used inside a module

@gavinbunney Confirm, in my case it does not work inside a module

[rguichard]

I think we hit the same issue with a for_each inside a module. I can confirm that it works outside a module but not inside.

kubectl 1.10.0
terraform 0.14.7

locals {
  apply = [for v in data.kubectl_file_documents.apply.documents : {           
    data : yamldecode(v)                                                         
    content : v                                                                  
    }                                                                            
  ] 
}
data "kubectl_file_documents" "apply" {                                                                           
  content = data.flux_install.main.content                                    
}                                                                                
                                                                                 
# Apply manifests on the cluster                                                 
resource "kubectl_manifest" "apply" {                                            
  for_each   = { for v in local.apply : lower(join("/", compact([v.data.apiVersion, v.data.kind, lookup(v.data.metadata, "namespace", ""), v.data.metadata.name]))) => v.content }
  depends_on = [kubernetes_namespace.fluxv2]                                     
  yaml_body  = each.value                                                        
} 

Error: Invalid for_each argument

  on .terraform/modules/addons/modules/scaleway/fluxv2.tf line 72, in resource "kubectl_manifest" "apply":
  72:   for_each   = { for v in local.apply : lower(join("/", compact([v.data.apiVersion, v.data.kind, lookup(v.data.metadata, "namespace", ""), v.data.metadata.name]))) => v.content }

The "for_each" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the for_each depends on.

Using -target to target the Kubernetes namespace solves the problem but we'd like to run only one apply for the whole state.

[cloudoutloud]

There is a workaround is this not the same as https://github.com/gavinbunney/terraform-provider-kubectl/issues/58

Although not ideal

[DanTulovsky]

I don't think this is the same. As mentioned in the original post, there are no variables being set here. I am hitting the same problem inside a module:

data "kubectl_path_documents" "manifests" {
  pattern = "${path.module}/yaml/k8s/*.yaml"
}

resource "kubectl_manifest" "vector-yaml" {
  count     = length(data.kubectl_path_documents.manifests.documents)
  yaml_body = element(data.kubectl_path_documents.manifests.documents, count.index)
}

gives:

Error: Invalid count argument

  on modules/vector/main.tf line 29, in resource "kubectl_manifest" "vector-yaml":
  29:   count     = length(data.kubectl_path_documents.manifests.documents)

The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.

% terraform version                                                                                               
Terraform v0.14.7
+ provider registry.terraform.io/gavinbunney/kubectl v1.10.0
+ provider registry.terraform.io/hashicorp/external v2.1.0
+ provider registry.terraform.io/hashicorp/google v3.58.0
+ provider registry.terraform.io/hashicorp/google-beta v3.58.0
+ provider registry.terraform.io/hashicorp/helm v2.0.2
+ provider registry.terraform.io/hashicorp/kubernetes v1.13.3
+ provider registry.terraform.io/hashicorp/kubernetes-alpha v0.2.1
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/mongey/kafka v0.2.12

[yongzhang]

I have the same issue and couldn't find any solution so far.

[bakayolo]

Having the same issue but with kubect_file_documents

data "kubectl_file_documents" "manifests" {
  content = file("${path.module}/manifests/metrics_server.yaml")
}

resource "kubectl_manifest" "metrics_server" {
  count     = length(data.kubectl_file_documents.manifests.documents)
  yaml_body = element(data.kubectl_file_documents.manifests.documents, count.index)
}

Resulting in

 6:   count     = length(data.kubectl_file_documents.manifests.documents)

The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.

[msevastian]

same issue

  50:   count     = length(data.kubectl_path_documents.manifests.documents)

The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.

[arodus]

I'm experiencing the same issue with the code found here: https://github.com/fluxcd/terraform-provider-flux/blob/main/examples/github/main.tf. Everything works fine but as soon as the code is put into a module planning fails with the following error:

local.sync will be known only after apply
 
The "for_each" value depends on resource attributes that cannot be
determined until apply, so Terraform cannot predict how many instances will
be created. To work around this, use the -target argument to first apply
only the resources that the for_each depends on.

[shyblower]

The same problem manifested itself here today. Yesterday and before it was working fine.

[Arsen-Uulu]

I had the same issue with vars. You can't reference asg_name = aws_autoscaliing_group.foo.name.

[theothermike]

Having same issue, only inside a module. TF 0.15.3 and 1.10.0 provider

[theothermike]

FYI I have a hackey workaround.

Basically, I create a submodule that is its own state, that all it does it use the flux provider to generate the Manifest YAML files, and generate some terraform source code that I use in the main flux module to install. This means since I am generating terraform source, the list of manifests is well known and wont throw this error. here's some snippets (repeat same code for sync). This ought to be workable for any manifests loaded with kubectl_file_documents

# Flux
data "flux_install" "main" {
  target_path    = local.target_path
  network_policy = false
}

data "kubectl_file_documents" "install" {
  content = data.flux_install.main.content
}

# Convert documents list to include parsed yaml data
locals {
  install = [for v in data.kubectl_file_documents.install.documents : {
    data : yamldecode(v)
    content : v
    }
  ]
  install_filenames = [for v in local.install : format("%s.yaml", replace(lower(join("/", compact([v.data.apiVersion, v.data.kind, lookup(v.data.metadata, "namespace", ""), v.data.metadata.name]))), "/", "-"))]
}

# generate the install manifest files
resource "local_file" "install" {
  for_each             = { for v in local.install : lower(join("/", compact([v.data.apiVersion, v.data.kind, lookup(v.data.metadata, "namespace", ""), v.data.metadata.name]))) => v.content }
  filename             = "${path.module}/../manifests/${var.environment}/${replace(each.key, "/", "-")}.yaml"
  content              = each.value
  file_permission      = "0644"
  directory_permission = "0755"
}

// generate TF that holds the list of manifests to work around https://github.com/gavinbunney/terraform-provider-kubectl/issues/61
resource "local_file" "install-tfcode" {
  filename             = "${path.module}/../flux_install_manifests-${var.environment}.tf"
  content              = <<TF
locals {
  flux_install_manifests_${var.environment} = [
    "${join("\",\n\t\"", local.install_filenames)}"
  ]
}
TF
  file_permission      = "0644"
  directory_permission = "0755"
}

then in the main flux module i can reference these YAML and locals

resource "kubectl_manifest" "install" {
  for_each   = var.environment == "development" ? toset(local.flux_install_manifests_development) : var.environment == "staging" ? toset(local.flux_install_manifests_staging) : var.environment == "production" ? toset(local.flux_install_manifests_production) : toset(local.flux_install_manifests_development)
  depends_on = [kubernetes_namespace.flux_system]
  yaml_body  = file("${path.module}/manifests/${var.environment}/${each.value}")
}

This will also serve as a nice way to upgrade the flux manifests when there is a new provider - just re-generate these files, and run terraform apply in the actual state

[vfrans]

Exact same problem here. Was working ysterday and now it fails with:

 The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the count depends on.

My code:

data "kubectl_path_documents" "kafka-cluster-manifests" {
  pattern = "${path.module}/kafka-cluster/*.yaml"
  disable_template = true
}
resource "kubectl_manifest" "kafka-cluster" {
  count     = length(data.kubectl_path_documents.kafka-cluster-manifests.documents)
  yaml_body = element(data.kubectl_path_documents.kafka-cluster-manifests.documents, count.index)
  wait = true
}

TF version:

francois@Francoiss-Mac-mini evs-ipdvia-aws % terraform version
Terraform v0.15.5
on darwin_arm64
+ provider registry.terraform.io/gavinbunney/kubectl v1.11.1
+ provider registry.terraform.io/hashicorp/aws v3.44.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.2.0
+ provider registry.terraform.io/hashicorp/helm v2.1.2
+ provider registry.terraform.io/hashicorp/kubernetes v1.13.4
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/terraform-aws-modules/http v2.4.1

[vfrans]

• exact same behaviour when I swap kubectl_path_documents to kubectl_filename_list

@gavinbunney are we doing something wrong? It seems to me like I really took the configration straight out of the documentation.

On top of that, it worked but now, with a clean state, terraform does not work because of that count...

[jjhidalgar]

Same issue here... Only happens if inside a module

I think I will put a file called "yaml_count.txt" and read it with tonumber(file("${module.path}/.../yaml_count.txt")) and then count all the individual yaml files and put the number there :) (until this is fixed!)

[hsalluri259]

Hi, I get the same error if I do terraform apply with no existing cluster. If I have a cluster then I don't get this error. It would be nice if terraform knows this to run with no existing cluster.

data "kubectl_path_documents" "controller_manifests" {
pattern = "${path.module}/files/controller_deploy.yaml"
vars = {
eks_cluster_name = var.eks_cluster_name
}
}
#Install the ALB controller
resource "kubectl_manifest" "install_controller_manifests" {
wait = true
count = length(data.kubectl_path_documents.controller_manifests.documents)
yaml_body = element(data.kubectl_path_documents.controller_manifests.documents, count.index)
depends_on = [kubectl_path_documents.controller_manifests]
}

when I run terraform apply first time, it throws the following error. The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the count depends on.

[g0atcheez]

Also having the same issue as others are experiencing using the same method that's indicated in the docs.

data "kubectl_file_documents" "certmanager_crds" {
  content = file("${path.module}/kubectl/certmanager-crds.yaml")
}

resource "kubectl_manifest" "certmanager-crds" {
  count      = length(data.kubectl_file_documents.certmanager_crds.documents)
  yaml_body  = element(data.kubectl_file_documents.certmanager_crds.documents, count.index)
  depends_on = [kubectl_manifest.rancher-import]
}

Error: Invalid count argument
│ 
│   on ../../Terraform.Modules.Azure.K8s-Base/src/main.tf line 179, in resource "kubectl_manifest" "certmanager-crds":
│  179:   count      = length(data.kubectl_file_documents.certmanager_crds.documents)
│ 
│ The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work
│ around this, use the -target argument to first apply only the resources that the count depends on.

❯ terraform version
Terraform v1.0.1
on darwin_amd64
+ provider registry.terraform.io/cloudflare/cloudflare v2.9.0
+ provider registry.terraform.io/gavinbunney/kubectl v1.11.3
+ provider registry.terraform.io/hashicorp/azuread v2.0.1
+ provider registry.terraform.io/hashicorp/google v3.81.0
+ provider registry.terraform.io/hashicorp/google-beta v3.81.0
+ provider registry.terraform.io/hashicorp/helm v2.0.3
+ provider registry.terraform.io/hashicorp/http v2.1.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.0.3
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/vault v2.23.0
+ provider registry.terraform.io/rancher/rancher2 v1.11.0

You can run terraform state show on the data resource and it returns the expected data so why would the plan say the attributes can't be determined until apply?

❯ terraform state show module.k8s-base.kubectl_manifest.certmanager-crds
# module.k8s-base.kubectl_manifest.certmanager-crds:
resource "kubectl_manifest" "certmanager-crds" {
    api_version             = "apiextensions.k8s.io/v1beta1"
    force_new               = false
    id                      = "/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/certificaterequests.cert-manager.io"
    kind                    = "CustomResourceDefinition"
...

[calexandre]

I'm having the same problem here inside a module. I can confirm that the hack provided here does not work either inside a module - I have not tested it outside of the module.

I even tested with a slimmed down version of a single yaml document with the following configuration and it still doesn't work...

data "kubectl_path_documents" "secrets_test_hack" {
  pattern = "${path.module}/k8s/secrets-test/00-namespace.yaml"
}

resource "kubectl_manifest" "secrets_test" {
  count     = length(data.kubectl_path_documents.secrets_test_hack.documents)
  yaml_body = element(data.kubectl_path_documents.secrets_test_hack.documents, count.index)
}

╷
│ Error: Invalid count argument
│ 
│   on modules/secrets-csi-test-app/main.tf line 28, in resource "kubectl_manifest" "secrets_test":
│   28:   count     = length(data.kubectl_path_documents.secrets_test_hack.documents)
│ 
│ The "count" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to
│ first apply only the resources that the count depends on.
╵

[calexandre]

I managed to build a hacky workaround that might work for most of you guys... Basically I replaced the count source using other terraform's built-in functions to fetch the document count...

Here's an example (taken from by previous comment:

data "kubectl_path_documents" "secrets_test" {
  pattern = "${path.module}/k8s/secrets-test/*.yaml"
}

resource "kubectl_manifest" "secrets_test" {
  count = length(
    flatten(
      toset([
        for f in fileset(".", data.kubectl_path_documents.secrets_test.pattern) : split("\n---\n", file(f))
        ]
      )
    )
  )
  yaml_body = element(data.kubectl_path_documents.secrets_test.documents, count.index)
}

What is happening here:

• I calculate all the possible documents in the same path that the data block is using using a fileset function
• I wrap that result in a for loop and to support the yaml's multi-document feature, I'm also splitting each file with \n---\n to ensure that I only grab the correct ones.
• In the end I must flatten the set because in case if multi-documents, there will be arrays inside arrays.

I know this is not an elegant solution and there might be other issues that I missed, but it's better than using -target to get around the issue...

I tested locally with multiple scenarios and for now this workaround works for me.

[jodem]

@calexandre I tested with terraform 1.0.5 and kubectl 1.11.3 on a fresh project and I don't reproduce this module issue.

data "kubectl_path_documents" "nginx_manifests" {
  pattern = "${path.module}/templates/deploy_nginx.yaml"
  vars = {
    domainName = var.domain_name
    namespace = var.name_space
  }
}

resource "kubectl_manifest" "deploy_nginx" {
  count     = length(data.kubectl_path_documents.nginx_manifests.documents)
  yaml_body = element(data.kubectl_path_documents.nginx_manifests.documents, count.index)
}

module "deploy_nginx_test1" {
  source = "./modules/nginx_deploy_module"
  domain_name = "test1"
  name_space = "spike-tf-test1"
}

Works in both creation and update.

[calexandre]

@jodem it doesn't happen always, that's why some folks are reporting the issue.. I've used this module for at least one year, and it never happened to me, until today...

[jodem]

Ok @calexandre thanks for the clarification, it's a bit scary. Thanks also for providing an elegant hack to mitigate the issue.

[calexandre]

Ok @calexandre thanks for the clarification, it's a bit scary. Thanks also for providing an elegant hack to mitigate the issue.

Thanks! I don't know about the "elegant" part ;) But the actual code, shouldn't be too different I guess...

[iJebus]

Yep, just started hitting this with no changes 😞. How odd.

[sidh]

Hit this bug today. Very annoying. :(

[luisdavim]

This is very strange, if I put this in a file on its own:

resource "kubectl_manifest" "pixie-viziers" {
  for_each  = data.kubectl_file_documents.pixie-viziers.manifests
  yaml_body = each.value
}

resource "kubectl_manifest" "pixie-crd" {
  for_each  = data.kubectl_file_documents.pixie-crd.manifests
  yaml_body = each.value
}

data "kubectl_file_documents" "pixie-viziers" {
  content = data.http.pixie-viziers.body
}

data "kubectl_file_documents" "pixie-crd" {
  content = data.http.pixie-crd.body
}

data "http" "pixie-viziers" {
  url = "https://raw.githubusercontent.com/pixie-labs/pixie/main/k8s/operator/crd/base/px.dev_viziers.yaml"
}

data "http" "pixie-crd" {
  url = "https://raw.githubusercontent.com/pixie-labs/pixie/main/k8s/operator/helm/crds/olm_crd.yaml"
}

provider "kubectl" {
  host                   = "host"
  cluster_ca_certificate = "cert"
  token                  = "token"
  load_config_file       = false
}

terraform {
  required_version = ">= 1.0.4"

  required_providers {
    kubectl = {
      source  = "gavinbunney/kubectl"
      version = ">= 1.13.1"
    }
    http = {
      source  = "hashicorp/http"
      version = "2.1.0"
    }
  }
}

it works but if I use that same code in a module, it fails with:

╷
│ Error: Invalid for_each argument
│
│   on .terraform/modules/eks/eks-init/newrelic.tf line 29, in resource "kubectl_manifest" "pixie-viziers":
│   29:   for_each  = data.kubectl_file_documents.pixie-viziers.manifests
│     ├────────────────
│     │ data.kubectl_file_documents.pixie-viziers.manifests is a map of string, known only after apply
│
│ The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the for_each depends on.
╵
╷
│ Error: Invalid for_each argument
│
│   on .terraform/modules/eks/eks-init/newrelic.tf line 34, in resource "kubectl_manifest" "pixie-crd":
│   34:   for_each  = data.kubectl_file_documents.pixie-crd.manifests
│     ├────────────────
│     │ data.kubectl_file_documents.pixie-crd.manifests is a map of string, known only after apply
│
│ The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the for_each depends on.
╵