`flux_bootstrap_git` do not deploy resources in cluster

[cebidhem]

Hey Flux team 👋

I tried to migrate Flux tf provider to 1.1.2 following the migration guide, and I updated the flux version to 2.0.0-rc5 but I have a weird issue: the manifests get pushed to the repo as expected, however I don't see any pods in the flux-system namespace anymore.

The terraform apply do not fail, and I've tried to rerun the bootstrap several times by updating the flux version (so that a change gets detected and the bootstrap rerun). So everytime the manifests get pushed to the repo, but no reconciliation is done because there's no pod to perform it.

At some point I thought maybe the use of provider.flux.kubernetes.token was the issue, so I tried as well with provider.flux.kubernetes.exec but it's the same result.

provider "flux" {
  kubernetes = {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
    exec = {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.cluster.id]
    }
  }
  git = {
    url = "ssh://[email protected]/MyGHOrg/${local.flux2.repository}.git"
    ssh = {
      username    = "git"
      private_key = try(tls_private_key.identity[0].private_key_pem, null)
    }
  }
}

I'm currently blocked at "dev" upgrade, but I can't really move to my other clusters if I don't understand this :(

I tried to look into the issues here, didn't find something similar. Any chance anyone could help with this please ?

The iam_role assumed to run the apply is part of "system:masters" and didn't change during migration.

Below the resource in the terraform state (remove most of the content for kustomization_override, repository_files.gotk-components.yaml and repository_files.gotk-sync.yaml for brevity):

{
      "mode": "managed",
      "type": "flux_bootstrap_git",
      "name": "flux",
      "provider": "provider[\"registry.terraform.io/fluxcd/flux\"]",
      "instances": [
        {
          "index_key": 0,
          "schema_version": 0,
          "attributes": {
            "cluster_domain": "cluster.local",
            "components": [
              "helm-controller",
              "kustomize-controller",
              "notification-controller",
              "source-controller"
            ],
            "components_extra": [
              "image-automation-controller",
              "image-reflector-controller"
            ],
            "disable_secret_creation": null,
            "id": "flux-system",
            "image_pull_secret": null,
            "interval": "1m0s",
            "kustomization_override": "apiVersion: kustomize.config.k8s.io/v1beta1\nkind: Kustomization\nresources:\n- gotk-components.yaml\n- gotk-sync.yaml\npatches:\n- patch: |-\n    apiVersion: kustomize.toolkit.fluxcd.io/v1\n    kind: Kustomization\n",
            "log_level": "info",
            "namespace": "flux-system",
            "network_policy": true,
            "path": "clusters/dev/aws-dev-us-east-1",
            "recurse_submodules": null,
            "registry": "ghcr.io/fluxcd",
            "repository_files": {
              "clusters/dev/aws-dev-us-east-1/flux-system/gotk-components.yaml": "---\n# This manifest was generated by flux. DO NOT EDIT.\n# Flux Version: v2.0.1\n# Components: helm-controller,kustomize-controller,notification-controller,source-controller,image-automation-controller,image-reflector-controller\n",
              "clusters/dev/aws-dev-us-east-1/flux-system/gotk-sync.yaml": "# This manifest was generated by flux. DO NOT EDIT.\n---"
            },
            "secret_name": "flux-system",
            "timeouts": null,
            "toleration_keys": null,
            "version": "v2.0.1",
            "watch_all_namespaces": true
          },
          "sensitive_attributes": [],
          "dependencies": [
            "data.aws_eks_cluster.cluster",
            "data.aws_eks_cluster_auth.cluster",
            "data.github_repository.main",
            "github_repository.main",
            "github_repository_deploy_key.main",
            "kubernetes_namespace.flux2",
            "tls_private_key.identity"
          ]
        }
      ]
    }

Thanks.