🐛 Wait and requeue if LB + its ports not deleted

[EmilienM]

What this PR does / why we need it:

Wait and requeue if LB is in PENDING_DELETE

If the LB that is being deleted when a cluster is deleted, it'll go through the PENDING_DELETE state and at this stage there is nothing we can do but wait for the LB to be actually deleted.

If the LB is in that state during the deletion, let's just return no error but request a reconcile after some time.

Wait for Octavia-managed ports to be removed

In a best-effort mode, when cleaning a load-balancer, wait for the ports with a device ID (mapped with the LB ID) and a certain prefix is deleted (by Octavia itself, not CAPO managed) before claiming the LB is really deleted.

This will avoid the reconcile to fail later when trying to remove the network while some ports are still attached.

Which issue(s) this PR fixes: Fixes #2124 Fixes #2121

[netlify[bot]]

✅ Deploy Preview for kubernetes-sigs-cluster-api-openstack ready!

Name	Link
🔨 Latest commit	7ef114bb9b11e089230fe33446aed7e0ad7c991a
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-cluster-api-openstack/deploys/670fc9870a30d10008a5364e
😎 Deploy Preview	https://deploy-preview-2122--kubernetes-sigs-cluster-api-openstack.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[EmilienM]

/hold I want to confirm we don't see the error in the logs.

[EmilienM]

/hold cancel I checked the logs and no more error of that type.

https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-openstack/2122/pull-cluster-api-provider-openstack-e2e-test/1800607078713135104/artifacts/clusters/bootstrap/logs/capo-system/capo-controller-manager/capo-controller-manager-7d957cb6cd-5dgc2/manager.log

[jichenjc]

/test pull-cluster-api-provider-openstack-e2e-test

[jichenjc]

/lgtm

[EmilienM]

@mdbooth ready for review when time permits. Maybe over-engineered but it does the job apparently. Feedback is open.

[EmilienM]

@mdbooth what to do with this one? It's been open for a long time.

[mdbooth]

What state did we leave it in? Is it about ready to go?

[EmilienM]

/hold cancel

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mdbooth

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• ~~OWNERS~~ [mdbooth]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment