kubernetes_manifest crashes during plan of argocd applicationset with matrix generator CR

[kacurez]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: v1.5.7
Kubernetes provider version: v2.30.0
Kubernetes version: v1.27

Affected Resource(s)

• kubernetes_manifest

Terraform Configuration Files

# manifest similar to https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Generators-Matrix/#overriding-parameters-from-one-child-generator-in-another-child-generator
resource "kubernetes_manifest" "application_set" {
  manifest = {
    apiVersion = "argoproj.io/v1alpha1"
    kind       = "ApplicationSet"
    metadata = {
      name      = "my-argocd-app-set"
      namespace = "argocd"
    }
    spec = {
      generators = [
        {
          matrix = {
            generators = [
              {
                git = {
                  repoURL         = "https://github.com/argoproj/argocd-example-apps"
                  revision        = "main"
                  pathParamPrefix = "firstApp"
                  files = [
                    {
                      path = "apps/templates/*.yaml"
                    }
                  ]
                }
              },
              {
                git = {
                  repoURL         = "https://github.com/argoproj/argocd-example-apps"
                  revision        = "main"
                  pathParamPrefix = "secondApp"
                  files = [
                    {
                      path = "apps/templates/*.yaml"
                    }
                  ]
                }
              }
            ]
          }
        }
      ]
      template = {
        metadata = {
          name = "{{firstApp.path.basename}}"
        }
        spec = {
          project = "default"
          source = {
            repoURL        = "https://github.com/argoproj/argocd-example-apps"
            targetRevision = "main"
          }
          destination = {
            server    = "https://kubernetes.default.svc"
            namespace = "default"
          }
        }
      }
    }
  }
}

Debug Output

https://gist.github.com/kacurez/d823f2de22766dbf8f4ba45df7a72447

Panic Output

https://gist.github.com/kacurez/d823f2de22766dbf8f4ba45df7a72447

Steps to Reproduce

1. install argocd e.g via helm https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd
2. define the kubernetes_manifest as specified above
3. terraform init
4. terraform plan

Expected Behavior

terraform plan doesn't crash and succeed to plan. Consequent terraform apply works as well by creating ApplicationSet CR

Actual Behavior

terraform plan crashes, see https://gist.github.com/kacurez/d823f2de22766dbf8f4ba45df7a72447

Important Factoids

If I run it with a single git generator then it works, for example:

      generators = [
        {
          matrix = {
            generators = [
              {
                git = {
                  repoURL         = "https://github.com/argoproj/argocd-example-apps"
                  revision        = "main"
                  pathParamPrefix = "secondApp"
                  files = [
                    {
                      path = "apps/templates/*.yaml"
                    }
                  ]
                }
              }
            ]
          }
        }
      ]

so the problem is most likely related to the generators array failing to proceed with 2 items.

References

• same issue but got closed https://github.com/hashicorp/terraform-provider-kubernetes/issues/1636

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[eli-nomasec]

Same here

[gablemire]

I'm having the same behaviour (with the same error stack) with version 2.31.0. When I apply the ApplicationSet manually through kubectl apply -f <my-appset.yaml>, it is successfully applied.

[ivankorn]

I logged similar issue, the only difference - I use the new fancy provider function

resource "kubernetes_manifest" "argocd_application_set" {
  manifest = provider::kubernetes::manifest_decode(file("argocd/application-set.yaml"))

  depends_on = [helm_release.chart]
}

If anyone is interested in temporary workaround for this issue, below is the code I use for that.

# Workaround for https://github.com/hashicorp/terraform-provider-kubernetes/issues/2580
module "kubectl-argocd-application-set" {
  source  = "terraform-google-modules/gcloud/google//modules/kubectl-wrapper"
  version = "~> 3.4"

  project_id        = local.gcp_project_id
  cluster_name      = module.gke.name
  cluster_location  = module.gke.location
  module_depends_on = [module.gke.endpoint]

  kubectl_create_command  = "kubectl create -f argocd/application-set.yaml"
  kubectl_destroy_command = "kubectl delete -f argocd/application-set.yaml"
  skip_download           = true
}

[github-actions[bot]]

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

[alejandro945]

Also here is another workaround for EKS

Workaround: Use gavinbunney/kubectl provider

Problem: kubernetes_manifest crashes with ArgoCD ApplicationSets using matrix generators with multiple child generators.

Solution: Replace kubernetes_manifest with kubectl_manifest from the gavinbunney/kubectl provider.

Setup:

# providers.tf
terraform {
  required_providers {
    kubectl = {
      source  = "gavinbunney/kubectl"
      version = "1.14.0"
    }
  }
}

provider "kubectl" {
  host                   = data.aws_eks_cluster.argocd.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.argocd.certificate_authority[0].data)
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.argocd.name]
  }
}

Usage:

# Instead of kubernetes_manifest (crashes)
resource "kubectl_manifest" "app_set" {
  yaml_body = templatefile("${path.module}/templates/app-set.yaml.tpl", {
    name = var.name
    namespace = var.namespace
  })
}

Status: Tested with complex matrix generators - works perfectly.