Reconciler error : "kustomization path not found" when removing namespace

[masterphenix]

Hello,

We have a git repository flux-workspaces that is organized in the following manner :

flux-workspaces
	<mycluster01>
		<namespace01>
			<tool01>
			<tool02>
			kustomization.yaml
			other_yaml_files.yaml
		<namespace02>
		<namespace03>
		...
	<mycluster02>
		...

The flux namespace is a bit different, as it has a kustomization resource for each namespace, that looks like this :

$ cat mycluster01/flux/fluxkustomization-mynamespace01.yaml
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: mycluster01-mynamespace01
spec:
  interval: 10m
  path: ./mycluster01/mynamespace01/
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-workspaces
  targetNamespace: mynamespace01

Each of these files is then referenced in the kustomization.yaml file of the flux namespace :

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- fluxkustomization-mynamespace01.yaml
...

So, to create a new namespace, we do the following :

• add a Kustomization for the syncing of the new namespace in the flux namespace : mycluster01/flux/fluxkustomization-mynamespace01.yaml
• reference that new file in mycluster01/flux/kustomization.yaml
• create the new namespace folder mycluster01/mynamespace01/ with the following items :
  • namespace.yaml : definition of the namespace
  • serviceaccount-default.yml : default service account
  • serviceaccount-jenkins.yml : jenkins service account
  • rolebinding-admin.yaml : bind the jenkins service account to an admin ClusterRole
  • kustomization.yaml : references the above files in its resources

After pushing these changes to the repo, everything is OK : the namespace and all its objects are created, and the logs are showing no errors.

Now, when we delete the namespace, the following is done :

• remove the namespace folder mycluster01/mynamespace01/
• remove the namespace kusto mycluster01/flux/fluxkustomization-mynamespace01.yaml
• remove the namespace kusto reference in the mycluster01/flux/kustomization.yaml resources

When these changes are commited and pushed to the repo :

• the namespace and all its objects are correctly removed from the cluster
• we get a whole bunch of errors in the kustomize controller logs :

{"level":"info","ts":"2021-01-08T08:22:16.756Z","logger":"controllers.Kustomization","msg":"Reconciliation finished in 999.976256ms, next run in 10m0s","controller":"kustomization","request":"flux/mycluster01-mynamespace01","revision":"master/28ff8ce47754604a8eeb9480a74b4568f78cc181"}
{"level":"error","ts":"2021-01-08T08:22:16.756Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"kustomize.toolkit.fluxcd.io","reconcilerKind":"Kustomization","controller":"kustomization","name":"mycluster01-mynamespace01","namespace":"flux","error":"kustomization path not found: stat /tmp/mycluster01-mynamespace01140601269/mycluster01/mynamespace01: no such file or directory"}
{"level":"info","ts":"2021-01-08T08:22:31.456Z","logger":"controllers.Kustomization","msg":"Reconciliation finished in 800.520349ms, next run in 10m0s","controller":"kustomization","request":"flux/mycluster01-mynamespace01","revision":"master/28ff8ce47754604a8eeb9480a74b4568f78cc181"}
{"level":"error","ts":"2021-01-08T08:22:31.456Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"kustomize.toolkit.fluxcd.io","reconcilerKind":"Kustomization","controller":"kustomization","name":"mycluster01-mynamespace01","namespace":"flux","error":"kustomization path not found: stat /tmp/mycluster01-mynamespace01296703260/mycluster01/mynamespace01: no such file or directory"}
{"level":"info","ts":"2021-01-08T08:22:32.164Z","logger":"controllers.Kustomization","msg":"Reconciliation finished in 696.992415ms, next run in 10m0s","controller":"kustomization","request":"flux/mycluster01-mynamespace01","revision":"master/28ff8ce47754604a8eeb9480a74b4568f78cc181"}
{"level":"error","ts":"2021-01-08T08:22:32.164Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"kustomize.toolkit.fluxcd.io","reconcilerKind":"Kustomization","controller":"kustomization","name":"mycluster01-mynamespace01","namespace":"flux","error":"kustomization path not found: stat /tmp/mycluster01-mynamespace01006052921/mycluster01/mynamespace01: no such file or directory"}
{"level":"info","ts":"2021-01-08T08:22:32.884Z","logger":"controllers.Kustomization","msg":"Reconciliation finished in 427.775447ms, next run in 10m0s","controller":"kustomization","request":"flux/mycluster01-mynamespace01","revision":"master/28ff8ce47754604a8eeb9480a74b4568f78cc181"}
{"level":"error","ts":"2021-01-08T08:22:32.884Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"kustomize.toolkit.fluxcd.io","reconcilerKind":"Kustomization","controller":"kustomization","name":"mycluster01-mynamespace01","namespace":"flux","error":"kustomization path not found: stat /tmp/mycluster01-mynamespace01924921258/mycluster01/mynamespace01: no such file or directory"}
{"level":"info","ts":"2021-01-08T08:22:33.386Z","logger":"controllers.Kustomization","msg":"Reconciliation finished in 221.770087ms, next run in 10m0s","controller":"kustomization","request":"flux/mycluster01-mynamespace01","revision":"master/28ff8ce47754604a8eeb9480a74b4568f78cc181"}
{"level":"error","ts":"2021-01-08T08:22:33.386Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"kustomize.toolkit.fluxcd.io","reconcilerKind":"Kustomization","controller":"kustomization","name":"mycluster01-mynamespace01","namespace":"flux","error":"kustomization path not found: stat /tmp/mycluster01-mynamespace01608322305/mycluster01/mynamespace01: no such file or directory"}
{"level":"info","ts":"2021-01-08T08:22:34.085Z","logger":"controllers.Kustomization","msg":"Reconciliation finished in 200.448415ms, next run in 10m0s","controller":"kustomization","request":"flux/mycluster01-mynamespace01","revision":"master/28ff8ce47754604a8eeb9480a74b4568f78cc181"}
{"level":"error","ts":"2021-01-08T08:22:34.085Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"kustomize.toolkit.fluxcd.io","reconcilerKind":"Kustomization","controller":"kustomization","name":"mycluster01-mynamespace01","namespace":"flux","error":"kustomization path not found: stat /tmp/mycluster01-mynamespace01093831276/mycluster01/mynamespace01: no such file or directory"}
{"level":"info","ts":"2021-01-08T08:22:34.603Z","logger":"controllers.Kustomization","msg":"Reconciliation finished in 216.664845ms, next run in 10m0s","controller":"kustomization","request":"flux/mycluster01-mynamespace01","revision":"master/28ff8ce47754604a8eeb9480a74b4568f78cc181"}
{"level":"error","ts":"2021-01-08T08:22:34.603Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"kustomize.toolkit.fluxcd.io","reconcilerKind":"Kustomization","controller":"kustomization","name":"mycluster01-mynamespace01","namespace":"flux","error":"kustomization path not found: stat /tmp/mycluster01-mynamespace01327836379/mycluster01/mynamespace01: no such file or directory"}
{"level":"info","ts":"2021-01-08T08:22:35.671Z","logger":"controllers.Kustomization","msg":"Reconciliation finished in 585.87802ms, next run in 10m0s","controller":"kustomization","request":"flux/mycluster01-mynamespace01","revision":"master/28ff8ce47754604a8eeb9480a74b4568f78cc181"}
{"level":"error","ts":"2021-01-08T08:22:35.671Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"kustomize.toolkit.fluxcd.io","reconcilerKind":"Kustomization","controller":"kustomization","name":"mycluster01-mynamespace01","namespace":"flux","error":"kustomization path not found: stat /tmp/mycluster01-mynamespace01296367742/mycluster01/mynamespace01: no such file or directory"}
{"level":"info","ts":"2021-01-08T08:22:36.579Z","logger":"controllers.Kustomization","msg":"Reconciliation finished in 267.677557ms, next run in 10m0s","controller":"kustomization","request":"flux/mycluster01-mynamespace01","revision":"master/28ff8ce47754604a8eeb9480a74b4568f78cc181"}
{"level":"error","ts":"2021-01-08T08:22:36.579Z","logger":"controller","msg":"Reconciler error","reconcilerGroup":"kustomize.toolkit.fluxcd.io","reconcilerKind":"Kustomization","controller":"kustomization","name":"mycluster01-mynamespace01","namespace":"flux","error":"kustomization path not found: stat /tmp/mycluster01-mynamespace01010473413/mycluster01/mynamespace01: no such file or directory"}
...
{"level":"info","ts":"2021-01-08T08:22:36.893Z","logger":"controllers.Kustomization","msg":"garbage collection completed: Kustomization/flux/mycluster01-mynamespace01 marked for deletion\n","kustomization":"flux/mycluster01-flux"}
{"level":"info","ts":"2021-01-08T08:22:36.941Z","logger":"controllers.Kustomization","msg":"Reconciliation finished in 16.388887738s, next run in 10m0s","controller":"kustomization","request":"flux/mycluster01-flux","revision":"master/28ff8ce47754604a8eeb9480a74b4568f78cc181"}
{"level":"info","ts":"2021-01-08T08:22:36.980Z","logger":"controllers.Kustomization","msg":"garbage collection completed: LimitRange/mynamespace01/mynamespace01-limit-range deleted\nServiceAccount/mynamespace01/default deleted\nServiceAccount/mynamespace01/jenkins deleted\nRoleBinding/mynamespace01/mynamespace01-admin deleted\nNamespace/mynamespace01 deleted\n","kustomization":"flux/mycluster01-mynamespace01"}

Any idea why we get these errors ?

We are using Kustomize Controller v0.5.3.