"prune: true" is ignored in Kustomizations applied by HelmRelease (regression)

[artem-nefedov]

Describe the bug

After upgrading Flux from 0.17.0 to 0.18.3, I've started observing a problem: objects installed by Kustomizations, which in turn were deployed by HelmRelease, are not deleted when Kustomization is deleted, even if prune: true is set (the Kustomization itself is deleted correctly).

I can also reproduce it locally with kubectl. If I deploy the following Kustomization, which creates a single namespace "foo":

apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: test
  namespace: default
spec:
  interval: 15m0s
  path: ./v/0.1.0/test
  prune: true
  sourceRef:
    kind: Bucket
    name: my-bucket
    namespace: flux-system

..and delete it like this: kubectl delete -f test.yaml, namespace "foo" will not be deleted. But, if I delete it like this: kubectl delete ks test, then namespace will be deleted. This makes me believe the issue is caused by recent changes related to server-side apply.

Steps to reproduce

1. Create helm chart that deploys Kustomizations (Kustomizations must have prune: true set)
2. Deploy chart with HelmRelease
3. Delete chart
4. Validate that Kustomization is deleted (it is)
5. Validate that objects deployed by Kustomization are deleted (they're not)

I specifically observed the problem with Namespaces created by Kustomizations.

Expected behavior

Everything is deleted.

Screenshots and recordings

No response

OS / Distro

N/A

Flux version

0.18.3

Flux check

✔ kubectl 1.21.3 >=1.18.0-0 ✔ Kubernetes 1.21.2-eks-0389ca3 >=1.16.0-0 ► private-repo/fluxcd/helm-controller:v0.12.0 ► private-repo/fluxcd/kustomize-controller:v0.15.5 ► private-repo/fluxcd/notification-controller:v0.17.1 ► private-repo/fluxcd/source-controller:v0.16.0

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[stefanprodan]

Can you please post here the namespace as it's left on the cluster after the HelmRelease deletion? kubectl get ns test -oyaml.

Also can you confirm that the Kustomization applied by the HelmRelease has prune: true in-cluster?

[artem-nefedov]

@stefanprodan Sure. Here's the info you asked, as well as full log of actions from scratch:

➜ k get ns foo
Error from server (NotFound): namespaces "foo" not found
➜ k get ks test
Error from server (NotFound): kustomizations.kustomize.toolkit.fluxcd.io "test" not found
➜ flux create hr test-hr --chart ./v/0.1.0/test-hr --source Bucket/my-bucket.flux-system -n default
✚ generating HelmRelease
► applying HelmRelease
✔ HelmRelease created
◎ waiting for HelmRelease reconciliation
✔ HelmRelease test-hr is ready
✔ applied revision 0.1.0
➜ k get ks test -o yaml | grep prune:
  prune: true
➜ k get ks test
NAME   READY   STATUS                                                       AGE
test   True    Applied revision: 340443ea2659db4c464bab5939dfacef817ba092   24s
➜ flux delete hr test-hr -n default -s
► deleting helmreleases test-hr in default namespace
✔ helmreleases deleted
➜ k get ks test
Error from server (NotFound): kustomizations.kustomize.toolkit.fluxcd.io "test" not found
➜ k get ns foo -o yaml
apiVersion: v1
kind: Namespace
metadata:
  creationTimestamp: "2021-10-16T07:58:26Z"
  labels:
    kubernetes.io/metadata.name: foo
    kustomize.toolkit.fluxcd.io/name: test
    kustomize.toolkit.fluxcd.io/namespace: default
  name: foo
  resourceVersion: "1587511"
  uid: 083a82f5-bdd7-4349-b903-756ce8457e45
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

[stefanprodan]

This is really strange, can you update the API version of your Kustomizations to v1beta2 and try again? Also if you can please post here k get ks test -oyaml.

[artem-nefedov]

@stefanprodan Thanks! Changing to v1beta2 actually solved the problem.

The weird thing is that, if my Kustomization manifest in helm chart says v1beta1, the manifest in cluster shows v1beta2, which is probably related to the problem:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  annotations:
    meta.helm.sh/release-name: test-hr
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2021-10-16T09:40:07Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    helm.toolkit.fluxcd.io/name: test-hr
    helm.toolkit.fluxcd.io/namespace: default
  name: test
  namespace: default
  resourceVersion: "1633920"
  uid: d539fc74-040c-451a-99f9-71f09567bdc1
spec:
  force: false
  interval: 15m0s
  path: ./v/0.1.0/test-ks
  prune: true
  sourceRef:
    kind: Bucket
    name: my-bucket
    namespace: flux-system
status:
  conditions:
  - lastTransitionTime: "2021-10-16T09:40:07Z"
    message: 'Applied revision: 340443ea2659db4c464bab5939dfacef817ba092'
    reason: ReconciliationSucceeded
    status: "True"
    type: Ready
  inventory:
    entries:
    - id: _foo__Namespace
      v: v1
  lastAppliedRevision: 340443ea2659db4c464bab5939dfacef817ba092
  lastAttemptedRevision: 340443ea2659db4c464bab5939dfacef817ba092
  observedGeneration: 1

And the same thing happens if I just apply Kustomization locally:

➜ cat ks.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1beta1
kind: Kustomization
metadata:
  name: test
  namespace: default
spec:
  interval: 15m0s
  path: ./v/0.1.0/test-ks
  prune: true
  sourceRef:
    kind: Bucket
    name: my-bucket
    namespace: flux-system

➜ k apply -f ks.yaml
kustomization.kustomize.toolkit.fluxcd.io/test created
➜ k get ks test -o yaml | grep apiVersion:
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2

As soon as I bumped version in manifest, it started to work correctly.

[stefanprodan]

IMO this is a bug in Helm itself, it should do the same thing as kubectl and delete objects using the API version as registered in-cluster instead of whatever is in the chart. Anyway this is not something we can fix in Flux.

[artem-nefedov]

@stefanprodan As I mentioned in my first post, the issue can be also reproduced with no helm at all, using only kubectl:

If I delete kustomization like this, namespace WILL NOT be deleted:

kubectl delete -f ks.yaml

If I instead delete it like this, namespace WILL be deleted:

kubectl delete ks test

My guess is, that is because in first case it also validates apiVersion match (which is different in-cluster and in manifest).

[jonathan-strzalka-saggs]

I am also seeing this but with children kustomizations(prune is true)

ex. parent kustomization creates child kustomization. I delete the parent, I expect the child to be deleted too. This worked in 18.x but it broke in 19.x

[stefanprodan]

@jonathan-strzalka-saggs have you bumped the API version of all your Kustomizations to v1beta2?

[jonathan-strzalka-saggs]

I have not. Trying now

edit: that did it! Thank you for the super quick response!

[takirala]

Does this mean using v1beta1 kustomization makes the garbage collection not work in latest flux?

I just tried upgrading from 0.17.2 -> 0.22.1 and after the bump, if i delete the kustomizations (v1beta1) manually, the resources created from that kustomization are still lying around. no errors in kustomize-controller logs and it does print that the garbage collection completed ..."msg":"garbage collection completed: Kustomization/.... message. Should i always use v1beta2 to make sure prune: true is honored?

When the kustomization is created with v1beta1, and since it is automatically stored as v1beta2 i do see the list of entries in the status.inventory field of kustomization. I am not sure why the resources are not being deleted by the controller.

When the kustomization is created with v1beta2, i see the same list of entries in status.inventory and i can also confirm that garbage collection is performed as expected. So creating v1beta2 kustomization works as expected but not v1beta1 kustomization. I have tested in flux 0.22.1

[hajdukda]

Should i always use v1beta2 to make sure prune: true is honored

This is super important question !!

[stefanprodan]

Should i always use v1beta2 to make sure prune: true is honored

Yes, v1beta1 was deprecated in favor of v1beta2 more than a year ago. Please upgrade all manifests with GitRepositories and Flux Kustomizations to v1beta2.