Upgrade to 2.2.3 not solve the issue: kustomization.yaml is empty

[khoinguyen]

Describe the bug

I upgraded to 2.2.3, but still get the same error (as described and closed in https://github.com/fluxcd/flux2/issues/4509): Logs:

kc logs kustomize-controller-6464685c8c-p9tdr | grep empty
{"level":"error","ts":"2024-02-12T05:55:53.572Z","msg":"Reconciliation failed after 10.825548072s, next try in 30m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"infra-unstable","namespace":"flux-system"},"namespace":"flux-system","name":"infra-unstable","reconcileID":"1ba7665d-f7d2-49ce-a6fe-2dcb1ad63725","revision":"develop@sha1:91026b76153cf514fd4ad29d3c88988cf6f91789","error":"kustomize build failed: kustomization.yaml is empty"}
{"level":"error","ts":"2024-02-12T05:56:07.288Z","msg":"Reconciliation failed after 10.707927756s, next try in 30m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"apps-unstable","namespace":"flux-system"},"namespace":"flux-system","name":"apps-unstable","reconcileID":"c0b921ab-72d8-408c-b97f-cf9fd54c208d","revision":"develop@sha1:91026b76153cf514fd4ad29d3c88988cf6f91789","error":"kustomize build failed: kustomization.yaml is empty"}
{"level":"error","ts":"2024-02-12T05:56:09.102Z","msg":"Reconciliation failed after 8.76190182s, next try in 30m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"infra-unstable","namespace":"flux-system"},"namespace":"flux-system","name":"infra-unstable","reconcileID":"db728136-7afc-48f9-9d16-61b199ec24f5","revision":"develop@sha1:91026b76153cf514fd4ad29d3c88988cf6f91789","error":"kustomize build failed: kustomization.yaml is empty"}
{"level":"error","ts":"2024-02-12T05:56:23.465Z","msg":"Reconciliation failed after 4.662710228s, next try in 30m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"apps-unstable","namespace":"flux-system"},"namespace":"flux-system","name":"apps-unstable","reconcileID":"4d9d3065-8f50-4094-ac28-ceb2556d968c","revision":"develop@sha1:91026b76153cf514fd4ad29d3c88988cf6f91789","error":"kustomize build failed: kustomization.yaml is empty"}
{"level":"error","ts":"2024-02-12T06:02:03.073Z","msg":"Reconciliation failed after 9.98762563s, next try in 30m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"apps-unstable","namespace":"flux-system"},"namespace":"flux-system","name":"apps-unstable","reconcileID":"c1214416-d58a-45ec-8d83-1b5d643afbf4","revision":"develop@sha1:691798195af8f47ba6254ddcc36b5fec7bb1cc45","error":"kustomize build failed: kustomization.yaml is empty"}
{"level":"error","ts":"2024-02-12T06:02:03.073Z","msg":"Reconciliation failed after 9.987724185s, next try in 30m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"infra-unstable","namespace":"flux-system"},"namespace":"flux-system","name":"infra-unstable","reconcileID":"9d7b9d04-65fe-4364-a535-1993bc4e7270","revision":"develop@sha1:691798195af8f47ba6254ddcc36b5fec7bb1cc45","error":"kustomize build failed: kustomization.yaml is empty"}
{"level":"error","ts":"2024-02-12T06:03:34.776Z","msg":"Reconciliation failed after 9.789412089s, next try in 30m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"apps-unstable","namespace":"flux-system"},"namespace":"flux-system","name":"apps-unstable","reconcileID":"85e4c6a5-f37e-408b-81fc-4381373cecc4","revision":"develop@sha1:8883bf3d29b13eb5c98147078d3bed26b75efa82","error":"kustomize build failed: kustomization.yaml is empty"}
{"level":"error","ts":"2024-02-12T06:03:34.877Z","msg":"Reconciliation failed after 9.891731856s, next try in 30m0s","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"infra-unstable","namespace":"flux-system"},"namespace":"flux-system","name":"infra-unstable","reconcileID":"3ff90165-8b87-43a2-adaa-8744bfbb1fb9","revision":"develop@sha1:8883bf3d29b13eb5c98147078d3bed26b75efa82","error":"kustomize build failed: kustomization.yaml is empty"}

Basically too avoid blocking in our system, the new charts or apps will be in *-unstable root kustomization, when it is stable, we move it to *-stable kustomization. This renders most of the time the *-unstable kustomization are empty.

Steps to reproduce

1. Upgrade to FluxCD 2.2.3, kustomize-controller upgraded to 1.2.2
2. Have following Kustomization definition:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infra-unstable
  namespace: flux-system
spec:
  interval: 30m0s
  path: ./infrastructure/kubernetes/infra/unstable
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system

and

apiVersion: kustomize.config.k8s.io/v1
kind: Kustomization
resources: []

3. Wait for FluxCD to changes

Expected behavior

Just silently skip the empty Kustomization

Screenshots and recordings

No response

OS / Distro

macOS sonoma (client)

Flux version

v2.2.3

Flux check

flux check ► checking prerequisites ✔ Kubernetes 1.26.12-eks-5e0fdde >=1.26.0-0 ► checking version in cluster ✔ distribution: flux-v2.2.3 ✔ bootstrapped: true ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.37.4 ✔ image-automation-controller: deployment ready ► ghcr.io/fluxcd/image-automation-controller:v0.37.1 ✔ image-reflector-controller: deployment ready ► ghcr.io/fluxcd/image-reflector-controller:v0.31.2 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v1.2.2 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v1.2.4 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v1.2.4 ► checking crds ✔ alerts.notification.toolkit.fluxcd.io/v1beta3 ✔ buckets.source.toolkit.fluxcd.io/v1beta2 ✔ gitrepositories.source.toolkit.fluxcd.io/v1 ✔ helmcharts.source.toolkit.fluxcd.io/v1beta2 ✔ helmreleases.helm.toolkit.fluxcd.io/v2beta2 ✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2 ✔ imagepolicies.image.toolkit.fluxcd.io/v1beta2 ✔ imagerepositories.image.toolkit.fluxcd.io/v1beta2 ✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1 ✔ kustomizations.kustomize.toolkit.fluxcd.io/v1 ✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2 ✔ providers.notification.toolkit.fluxcd.io/v1beta3 ✔ receivers.notification.toolkit.fluxcd.io/v1 ✔ all checks passed

Git provider

github

Container Registry provider

ECR

Additional context

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct