Fail to install velero helm chart 3.2.0. Getting error - job failed: BackoffLimitExceeded

[basaveswar-kureti]

What steps did you take and what happened: This has been working until Feb 19th, 2024.

kubectl create ns velero-320 helm upgrade --debug --install velero vmware-tanzu/velero --version 3.2.0 --namespace velero-320

What did you expect to happen: We have pinned to version 3.2.0 to avoid breakages and it has been working until yesterday.

The output of the following commands will help us better understand what's going on:

helm upgrade --debug --install velero vmware-tanzu/velero --version 3.2.0 --namespace velero-320
history.go:56: [debug] getting history for release velero
Release "velero" does not exist. Installing it now.
install.go:200: [debug] Original chart version: "3.2.0"
install.go:217: [debug] CHART PATH: /Users/kureti/Library/Caches/helm/repository/velero-3.2.0.tgz

client.go:134: [debug] creating 1 resource(s)
install.go:160: [debug] CRD backuprepositories.velero.io is already present. Skipping.
client.go:134: [debug] creating 1 resource(s)
install.go:160: [debug] CRD backups.velero.io is already present. Skipping.
client.go:134: [debug] creating 1 resource(s)
install.go:160: [debug] CRD backupstoragelocations.velero.io is already present. Skipping.
client.go:134: [debug] creating 1 resource(s)
install.go:160: [debug] CRD deletebackuprequests.velero.io is already present. Skipping.
client.go:134: [debug] creating 1 resource(s)
install.go:160: [debug] CRD downloadrequests.velero.io is already present. Skipping.
client.go:134: [debug] creating 1 resource(s)
install.go:160: [debug] CRD podvolumebackups.velero.io is already present. Skipping.
client.go:134: [debug] creating 1 resource(s)
install.go:160: [debug] CRD podvolumerestores.velero.io is already present. Skipping.
client.go:134: [debug] creating 1 resource(s)
install.go:160: [debug] CRD restores.velero.io is already present. Skipping.
client.go:134: [debug] creating 1 resource(s)
install.go:160: [debug] CRD schedules.velero.io is already present. Skipping.
client.go:134: [debug] creating 1 resource(s)
install.go:160: [debug] CRD serverstatusrequests.velero.io is already present. Skipping.
client.go:134: [debug] creating 1 resource(s)
install.go:160: [debug] CRD volumesnapshotlocations.velero.io is already present. Skipping.
client.go:478: [debug] Starting delete for "velero-upgrade-crds" ClusterRole
client.go:482: [debug] Ignoring delete failure for "velero-upgrade-crds" rbac.authorization.k8s.io/v1, Kind=ClusterRole: clusterroles.rbac.authorization.k8s.io "velero-upgrade-crds" not found
client.go:134: [debug] creating 1 resource(s)
client.go:478: [debug] Starting delete for "velero-server-upgrade-crds" ServiceAccount
client.go:482: [debug] Ignoring delete failure for "velero-server-upgrade-crds" /v1, Kind=ServiceAccount: serviceaccounts "velero-server-upgrade-crds" not found
client.go:134: [debug] creating 1 resource(s)
client.go:478: [debug] Starting delete for "velero-upgrade-crds" ClusterRoleBinding
client.go:482: [debug] Ignoring delete failure for "velero-upgrade-crds" rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding: clusterrolebindings.rbac.authorization.k8s.io "velero-upgrade-crds" not found
client.go:134: [debug] creating 1 resource(s)
client.go:478: [debug] Starting delete for "velero-upgrade-crds" Job
client.go:482: [debug] Ignoring delete failure for "velero-upgrade-crds" batch/v1, Kind=Job: jobs.batch "velero-upgrade-crds" not found
client.go:134: [debug] creating 1 resource(s)
client.go:706: [debug] Watching for changes to Job velero-upgrade-crds with timeout of 5m0s
client.go:734: [debug] Add/Modify event for velero-upgrade-crds: ADDED
client.go:773: [debug] velero-upgrade-crds: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
client.go:734: [debug] Add/Modify event for velero-upgrade-crds: MODIFIED
client.go:773: [debug] velero-upgrade-crds: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
client.go:734: [debug] Add/Modify event for velero-upgrade-crds: MODIFIED
Error: failed pre-install: 1 error occurred:
	* job failed: BackoffLimitExceeded


helm.go:84: [debug] failed pre-install: 1 error occurred:
	* job failed: BackoffLimitExceeded

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• helm version (use helm version): version.BuildInfo{Version:"v3.12.1", GitCommit:"f32a527a060157990e2aa86bf45010dfb3cc8b8d", GitTreeState:"clean", GoVersion:"go1.20.4"}
• helm chart version and app version (use helm list -n <YOUR NAMESPACE>):
• Kubernetes version (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration: GCP
• OS (e.g. from /etc/os-release):Mac or Linux