velero install failed due to velero-plugin-for-vsphere v1.3.1 is not working in tkgm CSI setup

[jhuisss]

Describe the bug

In my tkgm CSI setup with kubernetes version v1.22.8+vmware.1, I can not install velero server with velero-plugin-for-vsphere image v1.3.1, backup-driver pod and datamgr-for-vsphere-plugin pod are in error status due to: ... failed to retrieve velero-vsphere-plugin-config configuration, err: configmaps "velero-vsphere-plugin-config" not found. ... An error occurred: secrets "velero-vsphere-config-secret" not found ...

To Reproduce

velero install --provider aws --bucket velero --secret-file credentials-velero --snapshot-location-config region=minio --plugins velero/velero-plugin-for-aws:v1.1.0,vsphereveleroplugin/velero-plugin-for-vsphere:v1.3.1 --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://10.187.103.168:9000

Expected behavior

velero server installed successfully. All pods under velero ns are running or completed.

Troubleshooting Information

velero version

Client: Version: v1.7.0_vmware.1 Git commit: 9e52260568430ecb77ac38a677ce74267a8c2176 Server: ...

velero client config get features

features: <NOT SET>

velero-plugin-for-vsphere version: v1.3.1, and all other plugins in velero deployment: velero-plugin-for-aws:v1.1.0

Kubernetes cluster flavor: tkgm v1.22.8+vmware.1

vSphere CSI driver version: 2.1.0

helm list -A | grep vsphere-csi

vsphere-csi tca-system 1 2022-05-06 08:21:10.673453831 +0000 UTC deployed vsphere-csi-2.1.0 2.1.0

all images in vsphere-csi-controller deployment: Containers: csi-attacher: Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:a388619e471677fae9120e5169572f9b6f1532cb5cc608ab447fe3e2b43e628f csi-resizer: Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:9e62e1b29b9cb054a8e930a0615627ad1c4fbdf4b0c1aae58bd43489cc997f69 vsphere-csi-controller: Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:3eae8a0a84153978f2c85c4c4b387c9ef0380777f5c8d4d368e68083cf36d1ea liveness-probe: Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:f891e2d1871755b0e2f905052fc922fef02bdea3053df8e7d07062265214467c vsphere-syncer: Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:d68a71c0d87febdb1368fbebde07538f5e5438c459b018a7642d4c2de308a2ea csi-provisioner: Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:d939d67d10967b0b1fc4db7784238d5de2efdee1129ef2087534cb653cc6ef69

and all images of vsphere-csi-node daemonset: Containers: node-driver-registrar: Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:aaee5642312edc75df5ec8dd63d250f215233e3922d37c206b739dd4b0931928 vsphere-csi-node: Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:3eae8a0a84153978f2c85c4c4b387c9ef0380777f5c8d4d368e68083cf36d1ea liveness-probe: Image: projects.registry.vmware.com/tkg/packages/core/vsphere-csi@sha256:f891e2d1871755b0e2f905052fc922fef02bdea3053df8e7d07062265214467c

kubectl version

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8+vmware.1", GitCommit:"d797572df69c3951e4e8d495bf7720b594fd1c43", GitTreeState:"clean", BuildDate:"2022-03-21T23:22:25Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.8+vmware.1", GitCommit:"d797572df69c3951e4e8d495bf7720b594fd1c43", GitTreeState:"clean", BuildDate:"2022-03-21T23:17:28Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Screenshots

Anything else you would like to add:

I have compared source code of v1.1.1 and v1.3.1: https://github.com/vmware-tanzu/velero-plugin-for-vsphere/blob/8c3f94e55361fcc44898414a7ecb469822beecf9/pkg/utils/utils.go#L63 and https://github.com/vmware-tanzu/velero-plugin-for-vsphere/blob/f5fc6cf42a74278dd16719b330c8d40eca78b15b/pkg/utils/utils.go#L60. In the CSI setup, VC credential is stored as a secret under kube-system namespace, but in v1.3.1 it is not working, it is seeking for the configmap and secret under velero namespace. Seems it is a bug in v1.3.1.

[deepakkinni]

please follow instructions specified in https://github.com/vmware-tanzu/velero-plugin-for-vsphere/blob/main/docs/vanilla.md