The helm-delete job is not cleaned when helm release is deleted by helmchart CR

[GonzoHsu]

When I run kubectl delete helmchart <name> command, most of time I see the helm-delete- job is still existing, and it seems it is caused by duplicated helm-delete job created by helm-controller

For example, run kubectl delete helmchart nats,

The job and pod for helm delete are existing,

# kubectl get job | grep nats
helm-delete-nats                  1/1           3s         19m
# kubectl get pod | grep nats
helm-delete-nats-mn4xd                        0/1     Completed   0              19m

The event show the pod for /helm-delete-nats generated twice

# kubectl get event | grep nats
22m         Normal    SuccessfulCreate                  job/helm-delete-nats                    Created pod: helm-delete-nats-7h84g
22m         Normal    Scheduled                         pod/helm-delete-nats-7h84g              Successfully assigned default/helm-delete-nats-7h84g to 96e53ca49eb7e011eda21c000c290bc835
22m         Normal    Pulled                            pod/helm-delete-nats-7h84g              Container image "rancher/klipper-helm:v0.7.4-build20221121" already present on machine
22m         Normal    Created                           pod/helm-delete-nats-7h84g              Created container helm
22m         Normal    Started                           pod/helm-delete-nats-7h84g              Started container helm
22m         Normal    Killing                           pod/nats-0                              Stopping container nats
22m         Normal    Killing                           pod/nats-0                              Stopping container metrics
22m         Normal    Killing                           pod/nats-0                              Stopping container reloader
22m         Warning   CalculateExpectedPodCountFailed   poddisruptionbudget/nats                Failed to calculate the number of expected pods: statefulsets.apps does not implement the scale subresource
22m         Normal    Completed                         job/helm-delete-nats                    Job completed
22m         Normal    RemoveJob                         helmchart/nats                          Uninstalled HelmChart using Job default/helm-delete-nats, removing resources
21m         Normal    SuccessfulCreate                  job/helm-delete-nats                    Created pod: helm-delete-nats-mn4xd
21m         Normal    Scheduled                         pod/helm-delete-nats-mn4xd              Successfully assigned default/helm-delete-nats-mn4xd to 96e53ca49eb7e011eda21c000c290bc835
21m         Normal    Pulled                            pod/helm-delete-nats-mn4xd              Container image "rancher/klipper-helm:v0.7.4-build20221121" already present on machine
21m         Normal    Created                           pod/helm-delete-nats-mn4xd              Created container helm
21m         Normal    Started                           pod/helm-delete-nats-mn4xd              Started container helm
21m         Normal    Completed                         job/helm-delete-nats                    Job completed

And the existing pod logs are as following, I think it means the job does not find the helm release to delete.

# kubectl logs helm-delete-nats-mn4xd
if [[ ${KUBERNETES_SERVICE_HOST} =~ .*:.* ]]; then
	echo "KUBERNETES_SERVICE_HOST is using IPv6"
	CHART="${CHART//%\{KUBERNETES_API\}%/[${KUBERNETES_SERVICE_HOST}]:${KUBERNETES_SERVICE_PORT}}"
else
	CHART="${CHART//%\{KUBERNETES_API\}%/${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}}"
fi

set +v -x
+ [[ '' != \t\r\u\e ]]
+ export HELM_HOST=127.0.0.1:44134
+ HELM_HOST=127.0.0.1:44134
+ tiller --listen=127.0.0.1:44134 --storage=secret
+ helm_v2 init --skip-refresh --client-only --stable-repo-url https://charts.helm.sh/stable/
[main] 2023/03/06 09:51:27 Starting Tiller v2.17.0 (tls=false)
[main] 2023/03/06 09:51:27 GRPC listening on 127.0.0.1:44134
[main] 2023/03/06 09:51:27 Probes listening on :44135
[main] 2023/03/06 09:51:27 Storage driver is Secret
[main] 2023/03/06 09:51:27 Max history per release is 0
Creating /home/klipper-helm/.helm 
Creating /home/klipper-helm/.helm/repository 
Creating /home/klipper-helm/.helm/repository/cache 
Creating /home/klipper-helm/.helm/repository/local 
Creating /home/klipper-helm/.helm/plugins 
Creating /home/klipper-helm/.helm/starters 
Creating /home/klipper-helm/.helm/cache/archive 
Creating /home/klipper-helm/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://charts.helm.sh/stable/ 
Adding local repo with URL: http://127.0.0.1:8879/charts 
$HELM_HOME has been configured at /home/klipper-helm/.helm.
Not installing Tiller due to 'client-only' flag having been set
++ timeout -s KILL 30 helm_v2 ls --all '^nats$' --output json
++ jq -r '.Releases | length'
[storage] 2023/03/06 09:51:27 listing all releases with filter
+ V2_CHART_EXISTS=
+ [[ '' == \1 ]]
+ [[ '' == \v\2 ]]
+ [[ -f /config/ca-file.pem ]]
+ [[ -n '' ]]
+ shopt -s nullglob
+ helm_content_decode
+ set -e
+ ENC_CHART_PATH=/chart/nats.tgz.base64
+ CHART_PATH=/tmp/nats.tgz
+ [[ ! -f /chart/nats.tgz.base64 ]]
+ base64 -d /chart/nats.tgz.base64
+ CHART=/tmp/nats.tgz
+ set +e
+ [[ delete != \d\e\l\e\t\e ]]
+ helm_update delete
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
++ helm_v3 ls --all -f '^nats$' --namespace default --output json
++ jq -r '"\(.[0].app_version),\(.[0].status)"'
++ tr '[:upper:]' '[:lower:]'
+ LINE=null,null
+ IFS=,
+ read -r INSTALLED_VERSION STATUS _
+ VALUES=
+ for VALUES_FILE in /config/*.yaml
+ VALUES=' --values /config/values-01_HelmChart.yaml'
+ [[ delete = \d\e\l\e\t\e ]]
+ [[ -z null ]]
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
+ echo 'Uninstalling helm_v3 chart'
+ helm_v3 uninstall nats --namespace default
Error: uninstall: Release not loaded: nats: release: not found
+ true
+ exit