Use Equinix Metal LB service to provision the Kubernetes API VIP

[displague]

User Story

As a [developer/user/operator] I would like to have CAPP provision an IP via Metal LBaaS to create a Kubernetes API IP that will balance traffic between the control plane nodes. This would be a simpler to manage process than the existing K8s controller managing via KubeVIP or Metal API calls that juggle a Metal EIP.

Detailed Description

TODO: [A clear and concise description of what you want to happen.]

Anything else you would like to add:

TODO: [Miscellaneous information that will assist in solving the issue.]

• https://deploy.equinix.com/developers/docs/metal/networking/load-balancers/

/kind feature

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[cprivitere]

Current Status:

• [x] Add "EMLB" VIPManager type
• [x] Create emlb based template that creates a cluster with an EMLB-based VIP
• [x] Add LBaaS SDK to codebase (since it's not publically available yet)
• [x] Add emlb internal package
• [x] Can create new load balancer
• [x] Can create new origins
• [x] Can create new pools
• [x] Can create new listener ports
• [x] Can add origins to pools
• [x] Can add pool to listener port
• [x] Creates new load balancer, pool, port, and origin with new cluster using EMLB VIPManager type.
• [x] Deletes load balancer, pool, port, origin when deleting cluster.
• [x] Test coverage for new functions
• [x] Update e2e test to check emlb template
• [x] Update packetmachine and packetcluster to do the same things for kube-vip as they do for cpem, as it needs those as well.
• [x] ~Update packetmachine to handle worker node load balancer annotations.~ Exclude worker nodes from EMLB pool deletion processing.
• [x] Update e2e cleanup to clean up old emlbs too.
• [x] Update README to have an example in using EMLB