Error: Priority must be unique across all the request routing rules

[jkroepke]

trafficstars

Describe the bug

E0111 10:35:28.019949       1 controller.go:141] network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayRequestRoutingRulePriorityMustBeUnique" Message="Priority must be unique across all the request routing rules. Rules /subscriptions/3c668f12-0000-4844-861c-bf5364669149/resourceGroups/d-customer-app-rg/providers/Microsoft.Network/applicationGateways/d-customer-app-agw-01/requestRoutingRules/rr-f004bf78756cc24eb40a86bd77bd61d0 and /subscriptions/3c668f12-0000-4844-861c-bf5364669149/resourceGroups/d-customer-app-rg/providers/Microsoft.Network/applicationGateways/d-customer-app-agw-01/requestRoutingRules/rr-f09031d6b7be056016562d636185a3c2 cannot have the same priority 20000." Details=[]
E0111 10:35:28.019968       1 worker.go:62] Error processing event.network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayRequestRoutingRulePriorityMustBeUnique" Message="Priority must be unique across all the request routing rules. Rules /subscriptions/3c668f12-0000-4844-861c-bf5364669149/resourceGroups/d-customer-app-rg/providers/Microsoft.Network/applicationGateways/d-customer-app-agw-01/requestRoutingRules/rr-f004bf78756cc24eb40a86bd77bd61d0 and /subscriptions/3c668f12-0000-4844-861c-bf5364669149/resourceGroups/d-customer-app-rg/providers/Microsoft.Network/applicationGateways/d-customer-app-agw-01/requestRoutingRules/rr-f09031d6b7be056016562d636185a3c2 cannot have the same priority 20000." Details=[]

To Reproduce Creating >100 ingress objects in a row

Ingress Controller details

• Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list.
• Output of `kubectl logs .
• Any Azure support tickets associated with this issue.

Name:             agic-ingress-azure-7f6bf479b5-qptnd
Namespace:        agic
Priority:         0
Service Account:  agic-sa-ingress-azure
Node:             aks-user01-82964380-vmss00000d/172.24.0.4
Start Time:       Wed, 11 Jan 2023 10:04:40 +0100
Labels:           app=ingress-azure
                  pod-template-hash=7f6bf479b5
                  release=agic
Annotations:      checksum/config: 83f22312d0e6a21fbde286f3da7429f4ed7b79d725958909572ee9a6e4a7af00
                  cni.projectcalico.org/containerID: 778389587b6a30b18ac4a82e112ee58e2e0d2bb6609cafcfd88ca0d9d3af1188
                  cni.projectcalico.org/podIP: 100.127.0.67/32
                  cni.projectcalico.org/podIPs: 100.127.0.67/32
                  prometheus.io/port: 8123
                  prometheus.io/scrape: true
Status:           Running
IP:               100.127.0.67
IPs:
  IP:           100.127.0.67
Controlled By:  ReplicaSet/agic-ingress-azure-7f6bf479b5
Containers:
  ingress-azure:
    Container ID:   containerd://8e140638681bbbd42aaca5bd178001d8d0588b2e1f8c580e7ae5b745dba120bf
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.2
    Image ID:       mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:31a876143de3aca583f0508c0eb0d2a69e1d3da21dba003ca0bdfc2434f807bd
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 11 Jan 2023 10:04:44 +0100
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      agic-cm-ingress-azure  ConfigMap  Optional: false
    Environment:
      AZURE_CLOUD_PROVIDER_LOCATION:  /etc/appgw/azure.json
      AGIC_POD_NAME:                  agic-ingress-azure-7f6bf479b5-qptnd (v1:metadata.name)
      AGIC_POD_NAMESPACE:             agic (v1:metadata.namespace)
      AZURE_AUTH_LOCATION:            /etc/Azure/Networking-AppGW/auth/armAuth.json
    Mounts:
      /etc/Azure/Networking-AppGW/auth from networking-appgw-k8s-azure-service-principal-mount (ro)
      /etc/appgw/ from azure (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6gk9g (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  azure:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/
    HostPathType:  Directory
  networking-appgw-k8s-azure-service-principal-mount:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  networking-appgw-k8s-azure-service-principal
    Optional:    false
  kube-api-access-6gk9g:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                     Age                   From                       Message
  ----     ------                     ----                  ----                       -------
  Warning  FailedApplyingAppGwConfig  10s (x11 over 4m54s)  azure/application-gateway  network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayRequestRoutingRulePriorityMustBeUnique" Message="Priority must be unique across all the request routing rules. Rules /subscriptions/3c668f12-0000-4844-861c-bf5364669149/resourceGroups/d-customer-app-rg/providers/Microsoft.Network/applicationGateways/d-customer-app-agw-01/requestRoutingRules/rr-f004bf78756cc24eb40a86bd77bd61d0 and /subscriptions/3c668f12-0000-4844-861c-bf5364669149/resourceGroups/d-customer-app-rg/providers/Microsoft.Network/applicationGateways/d-customer-app-agw-01/requestRoutingRules/rr-f09031d6b7be056016562d636185a3c2 cannot have the same priority 20000." Details=[]

[olivercross]

Are you using the annotation appgw.ingress.kubernetes.io/ssl-redirect: "true" in your ingress definition by any chance?