Properly support updating vSphere credentials during upgrade

[jiayiwang7]

Currently, if you export the new vsphere credentials envs EKSA_VSPHERE_USERNAME and EKSA_VSPHERE_PASSWORD and run eksctl anywhere upgrade command, it fails at the preflight validation:

the VSphere credentials derived from VSPHERE_PASSWORD and VSPHERE_USERNAME are immutable; please use the same credentials for the upgraded cluster

In order to conduct the full credentials upgrade, user needs to manually update the vsphere_credentials in the cluster, and potentially update the cluster spec in order to trigger a full EKS-A CAPI cluster upgrade in order to apply the new credentials to all related secrets (e.g. cloud-provider-vsphere-credentials, CLUSTER_NAME-vsphere-credentials).

We need to have a better user experience around the credential updates through both CLI and FLC/controller, which might require redesigning some of the vsphere credentials logic in the codebase.

The goal is to improve the UX and be able to update those credentials in a simple and clear way. In order to achieve that, we need to review how EKS-A does the vSphere credential upgrade today, and re-design a better credential upgrade workflow with consideration of all the existing behavior across the board (management vs workload clusters, default credentials vs cloud provider credentials, upgrade through CLI vs FLC, etc.).

[jplewes]

If anyone can provide the steps necessary to manually update the vsphere_credentials in a cluster that would be very helpful. I have a need to change credentials and there is no information on how to do so.