CAPV should backoff vsphere when listing vsphere sessions

[jayunit100]

/kind bug

What steps did you take and what happened:

We had a cluster where the CAPV Account we were usign had limited "list" permissions. When trying to make a cluster, CAPV wound up trying to re-make a bunch of vsphere sessions and eventually exhausted the # of sessions. In some clusters this can overwhelm vsphere.

What did you expect to happen:

This wouldnt happen because, well... we dont want to overwhelm vsphere.

[warroyo]

The missing permission in vCenter that causes this is:

Sessions
 - Validate session

The error that is in the CAPV logs, when these permissions are missing is this:

E0516 20:14:39.844089       1 session.go:230] session "msg"="unable to get vim client session" "error"="ServerFaultCode: Permission to perform this operation was denied." 

[chrischdi]

Is this still applicable to the current version?

Maybe #2235 touches that area?

[chrischdi]

/assign @sbueringer

To get it on your radar :-)

[sbueringer]

/help

If someone has time to pick it up

[k8s-ci-robot]

@sbueringer: This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to this:

/help

If someone has time to pick it up

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sbueringer]

Related to #2235?

[sbueringer]

I'm pretty sure this should not happen anymore after Ricardo's PR #2530 (and maybe #2235)

@rikatz Can you take a quick look at this issue to check if it seems plausible that your PR resolved this?

(Although the change is basically logout if creating a client didn't work. Backoff is something that then happens by CR when bubbling up the error and the login consistently fails)