Disable vSphere CSI Driver

[jonathanmeier5]

Primary user story:

as a user i want to be able to disable the vsphere csi driver from being installed via my cluster spec

Additional user stories:

As a user, I would like to disable the vsphere csi driver from being managed by EKS-A in my existing clusters

Customer context: https://github.com/aws/eks-anywhere/issues/2908

This should be configurable via the cluster manifest definition file.

Open questions:

• should we automatically cleanup an existing vsphere CSI driver deployment if it's disabled on a cluster upgrade?

The templates are configured in this file.

[jonathanmeier5]

Discovery outcome on this ticket is as follows:

• we need to remove this ClusterResourceSet and everything it manages in order to disable CSI driver.
• we need to disable calls to this when the csi driver is disabled.
• we need to update vsphere.UpgradeNeeded to return true when we see a change in whatever config value we use to drive CSI driver.
• I'm pretty sure something in here has to change but I'm not sure what. Even when I commented out the ClusterResourceSet and its objects they were still recreated when a cluster upgrade finished.