Deleting an assigned managed identity blocks MIC pods

[Meandron]

Have you

• [x] Read Troubleshooting Guide
• [x] Read Known Issues
• [x] Searched on GitHub issues

Describe the bug We had to delete a number of managed identities in Azure and recreate them, but under a different resource group. During release of our application, this happens in a so called "infrastructure run". By that time, existing pods somehow seem to have that "old identites" assigned to them, leading to the situation that no new identites could be assigned to any pod anymore leading to deployment timeouts.

So, consider there is pod A) having a managed identity MI) assigned like /. Now, with the "infrastructure run", MI) gets deleted and MI2) gets created as <new_rg>/.

pod B) requesting a pod identity P) (different than MI) or MI2)) won't get the identity in time as the MIC pod somehow got completely blocked by the situation introduced above. Only workaround for us was to delete all existing pods logged by the mic pods, like:

failed to apply binding <namespace>/<pod>-job-identity-binding node and later resourcegroups/<old_rg>/providers/microsoft.managedidentity/userassignedidentities/<some_id> was not found

Steps To Reproduce

• Create Managed Identity
• Assign it as pod identity to a pod via MIC pods
• Delete the Managed Identity and recreate it under a new resource group
• Try to start a pod that needs a pod identity (= another user assigned managed identity)

Expected behavior

• MIC pods can handle that scenario, performing some sort of self-healing

AAD Pod Identity version

helm chart: aad-pod-identity-4.1.8

Kubernetes version

1.24.3

Additional context