Kubernetes Service with invalid settings generates an impossible to break loop trying to either create or delete the PIP in the LB

[djsly]

What happened:

We created a svc in kubernetes with bad annotation ( the ip-prefix-id was valid but the Kubernetes cluster didn't have access to it) This generated an error that was reoccurring every 5 minutes. Leavinghte Public IP creations in a pending state

E0810 02:03:40.625296       1 azure_loadbalancer.go:1594] reconcileLoadBalancer: failed to get load balancer for service "71bb6ffc-8447-4daf-b41a-aefcae6ff5b8/emissary", error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: {"error":{"code":"AuthorizationFailed","message":"The client '<redacted>' with object id '<redacted>' does not have authorization to perform action 'Microsoft.Network/publicIPAddresses/read' over scope '/subscriptions/<redacted>/resourceGroups/corp-cd1-network/providers/Microsoft.Network' or the scope is invalid. If access was recently granted, please refresh your credentials."}}

After we deleted the service in Kubernetes, we started getting this error instead.

I0810 04:54:05.172430       1 event.go:294] "Event occurred" object="71bb6ffc-8447-4daf-b41a-aefcae6ff5b8/emissary" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="DeleteLoadBalancerFailed" message="Error deleting load balancer: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 403, RawError: {\"error\":{\"code\":\"AuthorizationFailed\",\"message\":\"The client '<redacted>' with object id '<redacted>' does not have authorization to perform action 'Microsoft.Network/publicIPAddresses/read' over scope '/subscriptions/<redacted>/resourceGroups/corp-cd1-network/providers/Microsoft.Network' or the scope is invalid. If access was recently granted, please refresh your credentials.\"}}"

from this point on we cannot restore the state of the service unless we killl/restart the cloud-controller-manager

What you expected to happen:

I would expect that the cloud-controller-manager would not try to delete an IP of a service that was never provisioned, allows us to reuse the same service name when fixing the original use case.

How to reproduce it (as minimally and precisely as possible):

Set the following annotation, with a valid resourceID BUT one that the cloud controller manager doesnt have access to read

service.beta.kubernetes.io/azure-pip-prefix-id: /subscriptions/<redacted>

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"f18584a06fc476806da2e340e2eed960659871e8", GitTreeState:"clean", BuildDate:"2023-06-12T18:45:20Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration: AKS
• OS (e.g: cat /etc/os-release): N/A
• Kernel (e.g. uname -a): N/A
• Install tools: AKS
• Network plugin and version (I f this is a network-related bug):
• Others: