helm post-delete hooks is not working with argocd as expected

[dsharma24101992]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug I have an helm application present in path https://github.com/dsharma24101992/helmHooksDemo/tree/main/deployments/helm-hooks-demo which has 2 helm hooks . First one is pre-install hook which will create the clusterrole update-namespaces(Please refer to file https://github.com/dsharma24101992/helmHooksDemo/blob/main/deployments/helm-hooks-demo/templates/pre-install-hooks-job.yaml and job helm-hooks-demo-create-clusterrole-pre-install-job) . second is a post-delete hook.as part of post-delete hook(https://github.com/dsharma24101992/helmHooksDemo/blob/main/deployments/helm-hooks-demo/templates/post-delete-hooks-job.yaml), I am deleting the clusterrole update-namespaces as part of job delete-clusterrole). my understanding is that post-delete hooks for clusterrole, clusterrolebinding,serviceAccount should be be created first and as soon as my job delete-clusterrole is completed, the clusterrole ,clusterrolebinding,serviceAccount which are created as part of post-delete hooks should be deleted. but I am getting the following error when post-delete hooks are executing in argoUI.

clusterroles.rbac.authorization.k8s.io "helm-hooks-demo-post-delete" already exists

I am doing the same steps via helm install and uninstall, everything is working fine. below are the uninstall logs using helm uninstall.

deesharma@dac0272-lcedt:~/Downloads$ helm uninstall helm-hooks-demo -n helm-hooks-demo --debug 
uninstall.go:102: [debug] uninstall: Deleting helm-hooks-demo
client.go:486: [debug] Starting delete for "allow-all-traffic" NetworkPolicy
client.go:142: [debug] creating 1 resource(s)
client.go:142: [debug] creating 1 resource(s)
client.go:142: [debug] creating 1 resource(s)
client.go:142: [debug] creating 1 resource(s)
client.go:712: [debug] Watching for changes to Job delete-clusterrole with timeout of 5m0s
client.go:740: [debug] Add/Modify event for delete-clusterrole: ADDED
client.go:779: [debug] delete-clusterrole: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for delete-clusterrole: MODIFIED
client.go:779: [debug] delete-clusterrole: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for delete-clusterrole: MODIFIED
client.go:779: [debug] delete-clusterrole: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:740: [debug] Add/Modify event for delete-clusterrole: MODIFIED
client.go:486: [debug] Starting delete for "helm-hooks-demo-post-delete" ClusterRole
wait.go:66: [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:486: [debug] Starting delete for "helm-hooks-demo-post-delete" ServiceAccount
wait.go:66: [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:486: [debug] Starting delete for "helm-hooks-demo-post-delete" ClusterRoleBinding
wait.go:66: [debug] beginning wait for 1 resources to be deleted with timeout of 5m0s
client.go:486: [debug] Starting delete for "delete-clusterrole" Job
wait.go:66: [debug] beginning wait for 1 resources to be deleted with timeout of 
5m0s
uninstall.go:155: [debug] purge requested for helm-hooks-demo
release "helm-hooks-demo" uninstalled

I have attached the argoUI snapshot which I have taken after trigger application delete from argoUI.

helm-hooks-demo-UI-error during deletion

helm-hooks-demo-UI-install during application installation

In summary , the problem statement is why helm-hooks-demo-post-delete clusterrole is not getting deleted as soon as the post-delete job is completed.

To Reproduce

1. Git repo => https://github.com/dsharma24101992/helmHooksDemo.git
2. create the minikube cluster and install argo using command kubectl apply -n argocd https://raw.githubusercontent.com/argoproj/argo-cd/v2.10.10/manifests/install.yaml
3. Configure the helm-hooks-demo argo application using these 2 commands
   1. kubectl apply -f https://raw.githubusercontent.com/dsharma24101992/helmHooksDemo/main/helm-hooks-argo-application.yaml
   2. kubectl apply -f https://raw.githubusercontent.com/dsharma24101992/helmHooksDemo/main/helm-hooks-configmap.yaml
4. Delete the application from argoUI using foreground-deletion propogation policy.
5. The issue should be reproduced and the application deletion will throw the error clusterroles.rbac.authorization.k8s.io "helm-hooks-demo-post-delete" already exists

Expected behavior

Screenshots

Version

argocd: v2.5.8+bbe870f
  BuildDate: 2023-01-25T16:17:49Z
  GitCommit: bbe870ff5904dd1cebeba6c5dcb7129ce7c2b5e2
  GitTreeState: clean
  GoVersion: go1.18.10
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.5.8+bbe870f
  BuildDate: 2023-01-25T16:17:49Z
  GitCommit: bbe870ff5904dd1cebeba6c5dcb7129ce7c2b5e2
  GitTreeState: clean
  GoVersion: go1.18.10
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: could not get kustomize version: exec: "kustomize": executable file not found in $PATH
  Helm Version: v3.14.3+gf03cc04
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.18.0

[agaudreault]

Hey @dsharma24101992, can you add the relevant logs of your application for the application-controller component?

This might be related to https://github.com/argoproj/argo-cd/issues/18270

[dsharma24101992]

@agaudreault Yes, I also think the issue is similar. Please refer to the application-controller logs. Note that, the logs I am pasting is from the moment I tried the delete of the helm-hooks-demo application.

argocd-application-controller-logs.txt

[agaudreault]

Closed by https://github.com/argoproj/argo-cd/pull/18767. Feel free to reopen after you have tested with the code change if the bug is still happening.

[mboutet]

I'm still experiencing the issue on the following version:

{
    "Version": "v2.13.3+a25c8a0",
    "BuildDate": "2025-01-03T18:44:26Z",
    "GitCommit": "a25c8a0eef7830be0c2c9074c92dbea8ff23a962",
    "GitTreeState": "clean",
    "GoVersion": "go1.23.1",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.4.3 2024-07-19T16:40:33Z",
    "HelmVersion": "v3.15.4+gfa9efb0",
    "KubectlVersion": "v0.31.0",
    "JsonnetVersion": "v0.20.0"
}

I am seeing a very similar error to OP's:

clusterroles.rbac.authorization.k8s.io "some-operator-default-finalizer-cleanup" already exists

The resources that run after deletion in my chart are annotated with helm.sh/hook: post-delete only.

Here's my Application manifest:

project: some-operator.default
destination:
  server: https://kubernetes.default.svc
  namespace: some-operator-default
syncPolicy:
  automated:
    prune: true
    selfHeal: true
    allowEmpty: true
  syncOptions:
    - CreateNamespace=true
    - ServerSideApply=true
  retry:
    limit: 8
    backoff:
      duration: 5s
      factor: 2
      maxDuration: 2m
  managedNamespaceMetadata:
    labels:
      applications.argoproj.io/managed: 'true'
sources:
  - repoURL: REDACTED/charts/some-operator  # oci helm chart repo
    targetRevision: 3.1.4
    helm:
      valueFiles:
        - $values/operators/some-operator/instances/team-dev-local-1.default/values.yaml
      releaseName: some-operator-default
      ignoreMissingValueFiles: true
    chart: some-operator
  - repoURL: https://REDACTED  # github repo
    targetRevision: argocd-rwxfof
    ref: values
  - repoURL: https://REDACTED  # github repo
    path: argocd/operators-gen/team-dev-local-1
    targetRevision: argocd-rwxfof
    directory:
      recurse: true
      jsonnet: {}
      include: '{some-operator.default/*.yaml}'

Let me know if I should open a new issue instead. I replied to this one since it looks like the exact same problem.

[mboutet]

@agaudreault, sorry for the direct ping. Do you need any info from my end to help troubleshoot this?

[simonoff]

Hi, we have started to experience same issue. After updating helm the hooks not removed. Our installed version is v2.14.2