ApplicationGatewayILBDeploymentFailureDueToPrivateIPInUse

[galiacheng]

Describe the bug AGIC reported ApplicationGatewayILBDeploymentFailureDueToPrivateIPInUse error intermittently when enabling private IP. The application gateway v2 was deployed with public and private frontend IP. The frontend IP is in the same subnet of application gateway.

To Reproduce

This is an intermittent issue, not 100% reproduced.

Steps to reproduce the behavior:

1. Deploy Application Gateway V2
   • Frontends IP address type: Both
   • Public IP: add new
   • Private IP address: use static IP, pick one of available IP from the application gateway subnet
2. Deploy AKS, use kubenet
3. Peer AKS VNET and Application Gateway VNET, following How to setup networking between Application Gateway and AKS -Deployed in different vnets
4. Deploy AGIC 1.5.1 with Helm and enable private IP, configuration like:

   # This file contains the essential configs for the ingress controller Helm chart
   
   # Verbosity level of the App Gateway Ingress Controller
   verbosityLevel: 3
   
   ################################################################################
   # Specify which application gateway the ingress controller will manage
   #
   appgw:
       subscriptionId: @SUB_ID@
       resourceGroup: @APPGW_RG_NAME@
       name: @APPGW_NAME@
       usePrivateIP: true
   
       # Setting appgw.shared to "true" will create an AzureIngressProhibitedTarget CRD.
       # This prohibits AGIC from applying config for any host/path.
       # Use "kubectl get AzureIngressProhibitedTargets" to view and change this.
       shared: false
   
   ################################################################################
   # Specify which Kubernetes namespace the ingress controller will watch
   # Default value is "default"
   # Leaving this variable out or setting it to blank or empty string would
   # result in ingress controller observing all acessible namespaces.
   #
   kubernetes:
     watchNamespace: default
   
   ################################################################################
   # Specify the authentication with Azure Resource Manager
   #
   # Two authentication methods are available:
   # - Option 1: AAD-Pod-Identity (https://github.com/Azure/aad-pod-identity)
   # armAuth:
   #    type: aadPodIdentity
   #    identityResourceID: 
   #    identityClientID: 
   
   armAuth:
       type: servicePrincipal
       secretJSON: @SP_ENCODING_CREDENTIALS@
   
   ################################################################################
   # Specify if the cluster is RBAC enabled or not
   rbac:
     # Specifies whether RBAC resources should be created
     create: true
   
   

5. Deploy a sample application to default namespace and create ingress

Ingress Controller details

Error from AGIC:

E0602 10:15:35.864328       1 controller.go:141] Code="ApplicationGatewayILBDeploymentFailureDueToPrivateIPInUse" Message="Unable to reserve private IP '172.16.0.4', as it is already in use. Please select a different private IP." Details=[]
E0602 10:15:35.864350       1 worker.go:62] Error processing event.Code="ApplicationGatewayILBDeploymentFailureDueToPrivateIPInUse" Message="Unable to reserve private IP '172.16.0.4', as it is already in use. Please select a different private IP." Details=[]

Check the IP address, it's used by Application Gateway VMSS.

az network vnet check-ip-address \
>         --ids /subscriptions/1970354a-e600-4847-8848-89ea15c7a8c3/resourceGroups/jiangma-liberty-aks-test/providers/Microsoft.Network/virtualNetworks/olaks-app-gateway-vnet/subnets/olaks-gateway-subnet \
>         --ip-address 172.16.0.4
{
  "available": false,
  "availableIpAddresses": [
    "172.16.0.5",
    "172.16.0.6",
    "172.16.0.8",
    "172.16.0.9",
    "172.16.0.10"
  ],
  "inUseWithResource": "/subscriptions/9b06883d-b117-4c88-8a73-de5dc6cdd84b/resourceGroups/armrg-df62de59-e5c9-4f1b-ba40-82274bd5123d/providers/Microsoft.Network/networkInterfaces/|providers|Microsoft.Compute|virtualMachineScaleSets|appgw|virtualMachines|0|networkInterfaces|custnic",
  "isPlatformReserved": false
}

[galiacheng]

Support number: 2207190030000238