Redpanda Operator spams with sh.helm.release secrets when workload unhealthy

[jirislav]

When deployment unhealthy, the operator creates many secret objects:

sh.helm.release.v1.redpanda.v100                     helm.sh/release.v1   1      82m
sh.helm.release.v1.redpanda.v101                     helm.sh/release.v1   1      67m
sh.helm.release.v1.redpanda.v102                     helm.sh/release.v1   1      51m
sh.helm.release.v1.redpanda.v103                     helm.sh/release.v1   1      36m
sh.helm.release.v1.redpanda.v104                     helm.sh/release.v1   1      21m
sh.helm.release.v1.redpanda.v105                     helm.sh/release.v1   1      6m3s
sh.helm.release.v1.redpanda.v2                       helm.sh/release.v1   1      14d
sh.helm.release.v1.redpanda.v31                      helm.sh/release.v1   1      18h
sh.helm.release.v1.redpanda.v32                      helm.sh/release.v1   1      18h
sh.helm.release.v1.redpanda.v33                      helm.sh/release.v1   1      18h
sh.helm.release.v1.redpanda.v34                      helm.sh/release.v1   1      18h
sh.helm.release.v1.redpanda.v35                      helm.sh/release.v1   1      17h
sh.helm.release.v1.redpanda.v36                      helm.sh/release.v1   1      17h
sh.helm.release.v1.redpanda.v37                      helm.sh/release.v1   1      17h
sh.helm.release.v1.redpanda.v38                      helm.sh/release.v1   1      17h
sh.helm.release.v1.redpanda.v39                      helm.sh/release.v1   1      16h
sh.helm.release.v1.redpanda.v40                      helm.sh/release.v1   1      16h
sh.helm.release.v1.redpanda.v41                      helm.sh/release.v1   1      16h
sh.helm.release.v1.redpanda.v42                      helm.sh/release.v1   1      16h
sh.helm.release.v1.redpanda.v43                      helm.sh/release.v1   1      15h
sh.helm.release.v1.redpanda.v44                      helm.sh/release.v1   1      15h
sh.helm.release.v1.redpanda.v45                      helm.sh/release.v1   1      15h
sh.helm.release.v1.redpanda.v46                      helm.sh/release.v1   1      15h
sh.helm.release.v1.redpanda.v47                      helm.sh/release.v1   1      14h
sh.helm.release.v1.redpanda.v48                      helm.sh/release.v1   1      14h
sh.helm.release.v1.redpanda.v49                      helm.sh/release.v1   1      14h
sh.helm.release.v1.redpanda.v50                      helm.sh/release.v1   1      14h
sh.helm.release.v1.redpanda.v51                      helm.sh/release.v1   1      13h
sh.helm.release.v1.redpanda.v52                      helm.sh/release.v1   1      13h
sh.helm.release.v1.redpanda.v53                      helm.sh/release.v1   1      13h
sh.helm.release.v1.redpanda.v54                      helm.sh/release.v1   1      13h
sh.helm.release.v1.redpanda.v55                      helm.sh/release.v1   1      12h
sh.helm.release.v1.redpanda.v56                      helm.sh/release.v1   1      12h
sh.helm.release.v1.redpanda.v57                      helm.sh/release.v1   1      12h
sh.helm.release.v1.redpanda.v58                      helm.sh/release.v1   1      12h
sh.helm.release.v1.redpanda.v59                      helm.sh/release.v1   1      11h
sh.helm.release.v1.redpanda.v60                      helm.sh/release.v1   1      11h
sh.helm.release.v1.redpanda.v61                      helm.sh/release.v1   1      11h
sh.helm.release.v1.redpanda.v62                      helm.sh/release.v1   1      11h
sh.helm.release.v1.redpanda.v63                      helm.sh/release.v1   1      10h
sh.helm.release.v1.redpanda.v64                      helm.sh/release.v1   1      10h
sh.helm.release.v1.redpanda.v65                      helm.sh/release.v1   1      10h
sh.helm.release.v1.redpanda.v66                      helm.sh/release.v1   1      10h
sh.helm.release.v1.redpanda.v67                      helm.sh/release.v1   1      9h
sh.helm.release.v1.redpanda.v68                      helm.sh/release.v1   1      9h
sh.helm.release.v1.redpanda.v69                      helm.sh/release.v1   1      9h
sh.helm.release.v1.redpanda.v70                      helm.sh/release.v1   1      8h
sh.helm.release.v1.redpanda.v71                      helm.sh/release.v1   1      8h
sh.helm.release.v1.redpanda.v72                      helm.sh/release.v1   1      8h
sh.helm.release.v1.redpanda.v73                      helm.sh/release.v1   1      8h
sh.helm.release.v1.redpanda.v74                      helm.sh/release.v1   1      7h59m
sh.helm.release.v1.redpanda.v75                      helm.sh/release.v1   1      7h43m
sh.helm.release.v1.redpanda.v76                      helm.sh/release.v1   1      7h28m
sh.helm.release.v1.redpanda.v77                      helm.sh/release.v1   1      7h13m
sh.helm.release.v1.redpanda.v78                      helm.sh/release.v1   1      6h58m
sh.helm.release.v1.redpanda.v79                      helm.sh/release.v1   1      6h42m
sh.helm.release.v1.redpanda.v80                      helm.sh/release.v1   1      6h27m
sh.helm.release.v1.redpanda.v81                      helm.sh/release.v1   1      6h12m
sh.helm.release.v1.redpanda.v82                      helm.sh/release.v1   1      5h57m
sh.helm.release.v1.redpanda.v83                      helm.sh/release.v1   1      5h41m
sh.helm.release.v1.redpanda.v84                      helm.sh/release.v1   1      5h26m
sh.helm.release.v1.redpanda.v85                      helm.sh/release.v1   1      5h11m
sh.helm.release.v1.redpanda.v86                      helm.sh/release.v1   1      4h56m
sh.helm.release.v1.redpanda.v87                      helm.sh/release.v1   1      4h40m
sh.helm.release.v1.redpanda.v88                      helm.sh/release.v1   1      4h25m
sh.helm.release.v1.redpanda.v89                      helm.sh/release.v1   1      4h10m
sh.helm.release.v1.redpanda.v90                      helm.sh/release.v1   1      3h55m
sh.helm.release.v1.redpanda.v91                      helm.sh/release.v1   1      3h39m
sh.helm.release.v1.redpanda.v92                      helm.sh/release.v1   1      3h24m
sh.helm.release.v1.redpanda.v93                      helm.sh/release.v1   1      3h9m
sh.helm.release.v1.redpanda.v94                      helm.sh/release.v1   1      174m
sh.helm.release.v1.redpanda.v95                      helm.sh/release.v1   1      158m
sh.helm.release.v1.redpanda.v96                      helm.sh/release.v1   1      143m
sh.helm.release.v1.redpanda.v97                      helm.sh/release.v1   1      128m
sh.helm.release.v1.redpanda.v98                      helm.sh/release.v1   1      112m
sh.helm.release.v1.redpanda.v99                      helm.sh/release.v1   1      97m

Steps to reproduce:

1. Create a healthy setup
2. Make the Statefulset fail scheduling new PODs, e.g. by kubectl label namespace NAMESPACE pod-security.kubernetes.io/enforce=baseline
3. Delete the running PODs & either force rollout restart of the STS, or wait for the operator to do it.

JIRA Link: K8S-194