ApplicationGatewayPathsWithinUrlPathMapMustBeUnique when upgrading AGIC to 1.5 rc1

[DavidJohnsonSN]

trafficstars

Describe the bug When updating to AGIC 1.5.rc-1 we came across the following issue. Worth saying that up to this point our AGIC works fine with our setup and we have no problems deploying. Following the update we saw the error ApplicationGatewayPathsWithinUrlPathMapMustBeUnique in the logs and AGIC can no longer update the gateway and we believe this is due to a domain being shared with non-AGIC resources

Here is our setup in the gateway: www.domain.com/path-to-service-1: Backend pool is some VMs running in Azure www.domain.com/path-to-service-2: Backend pool is some VMs running in Azure www.domain.com/path-to-service-3: App running in our cluster managed by AGIC www.domain.com/path-to-service-4: App running in our cluster managed by AGIC www.a-n-otherdomain.com/path-to-service5: App running in our cluster managed by AGIC

The logs indicate that when AGIC tries to update the rules it says it cannot do so because path-to-service-3 is already defined in another PathRule. It seems to be trying to create it again rather than realise it already existed and should just update.

We also noticed it tried to recreate some of the configuration that already exists but i am unsure if that is related to this problem.

Here is the output from our logs which is the first update it tries to do after updating to 1.5.rc1

I1110 11:22:56.954946       1 pools.go:69] [brownfield] Existing Pools AGIC will remove: n/a
I1110 11:22:56.955877       1 listeners.go:103] [brownfield] Listeners AGIC created: fl-56495799ebea9e258a991f84c79c3e9d, fl-0cded256f4e4e25e1eff03a0411e18f0, fl-78770be29495f3e6f934bfd44a584b3a, fl-0489fe866c0a56eb6beff3a66bbb70f1, fl-566d8706e465ae7035ecd1574ff5e21b, fl-eca4cef607966e0049b0ccf6cf643258
I1110 11:22:56.955912       1 listeners.go:104] [brownfield] Existing Blacklisted Listeners AGIC will retain: fl-9820740dd5f29cc303e708396479c2ed, mee-services-listener
I1110 11:22:56.955920       1 listeners.go:105] [brownfield] Existing Listeners AGIC will remove: fl-e1903c8aa3446b7b3207aec6d6ecba8a
E1110 11:22:56.956045       1 routing_rules.go:166] Code="ErrorGeneratingListeners" Message="[brownfield] Could not find listener fl-e1903c8aa3446b7b3207aec6d6ecba8a in index"
E1110 11:22:56.956140       1 routing_rules.go:189] [brownfield] Could not obtain hostname for rule rr-e1903c8aa3446b7b3207aec6d6ecba8a; Skipping rule
E1110 11:22:56.956744       1 routing_rules.go:166] Code="ErrorGeneratingListeners" Message="[brownfield] Could not find listener fl-e1903c8aa3446b7b3207aec6d6ecba8a in index"
E1110 11:22:56.956801       1 routing_rules.go:189] [brownfield] Could not obtain hostname for rule rr-e1903c8aa3446b7b3207aec6d6ecba8a; Skipping rule
I1110 11:22:56.957028       1 redirects.go:52] [brownfield] Redirects AGIC created: n/a
I1110 11:22:56.957083       1 redirects.go:53] [brownfield] Existing Blacklisted Redirects AGIC will retain: n/a
I1110 11:22:56.957115       1 redirects.go:54] [brownfield] Existing Redirects AGIC will remove: n/a
E1110 11:22:56.959176       1 routing_rules.go:166] Code="ErrorGeneratingListeners" Message="[brownfield] Could not find listener fl-e1903c8aa3446b7b3207aec6d6ecba8a in index"
E1110 11:22:56.959199       1 routing_rules.go:189] [brownfield] Could not obtain hostname for rule rr-e1903c8aa3446b7b3207aec6d6ecba8a; Skipping rule
I1110 11:22:56.959301       1 pathmaps.go:93] [brownfield] PathMaps AGIC created: url-0489fe866c0a56eb6beff3a66bbb70f1
I1110 11:22:56.959313       1 pathmaps.go:94] [brownfield] Existing Blacklisted PathMaps AGIC will retain: mee-services-routing
I1110 11:22:56.959319       1 pathmaps.go:95] [brownfield] Existing PathMaps AGIC will remove: n/a
E1110 11:22:56.959363       1 routing_rules.go:166] Code="ErrorGeneratingListeners" Message="[brownfield] Could not find listener fl-e1903c8aa3446b7b3207aec6d6ecba8a in index"
E1110 11:22:56.959374       1 routing_rules.go:189] [brownfield] Could not obtain hostname for rule rr-e1903c8aa3446b7b3207aec6d6ecba8a; Skipping rule
I1110 11:22:56.959644       1 routing_rules.go:95] [brownfield] Rules AGIC created: rr-78770be29495f3e6f934bfd44a584b3a, rr-0489fe866c0a56eb6beff3a66bbb70f1, rr-566d8706e465ae7035ecd1574ff5e21b, rr-eca4cef607966e0049b0ccf6cf643258, rr-56495799ebea9e258a991f84c79c3e9d, rr-0cded256f4e4e25e1eff03a0411e18f0
I1110 11:22:56.959656       1 routing_rules.go:96] [brownfield] Existing Blacklisted Rules AGIC will retain: mee-services-routing
I1110 11:22:56.959663       1 routing_rules.go:97] [brownfield] Existing Rules AGIC will remove: rr-e1903c8aa3446b7b3207aec6d6ecba8a
I1110 11:22:56.979783       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I1110 11:22:57.480889       1 mutate_app_gateway.go:183] END AppGateway deployment
E1110 11:22:57.481020       1 controller.go:141] network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayPathsWithinUrlPathMapMustBeUnique" Message="Paths specified in a UrlPathMap must be unique. Path /external-class-management/* is already specified in another PathRule in UrlPathMap /subscriptions/9defd5ce-87f9-40ce-bfac-19f5217afc5b/resourceGroups/MEE-PROD-RG/providers/Microsoft.Network/applicationGateways/MEE-PROD-AGW/urlPathMaps/mee-services-routing." Details=[]
E1110 11:22:57.481506       1 worker.go:62] Error processing event.network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayPathsWithinUrlPathMapMustBeUnique" Message="Paths specified in a UrlPathMap must be unique. Path /external-class-management/* is already specified in another PathRule in UrlPathMap /subscriptions/9defd5ce-87f9-40ce-bfac-19f5217afc5b/resourceGroups/MEE-PROD-RG/providers/Microsoft.Network/applicationGateways/MEE-PROD-AGW/urlPathMaps/mee-services-routing." Details=[]

Our pod description

Name:         prod-ingress-ingress-azure-7d5ccdfb9b-9zlbc
Namespace:    ingress
Priority:     0
Node:         aks-agentpool-35064155-vmss000001/10.90.1.163
Start Time:   Wed, 10 Nov 2021 11:04:53 +0000
Labels:       app=ingress-azure
              pod-template-hash=7d5ccdfb9b
              release=prod-ingress
Annotations:  checksum/config: 9fb36437bb2be0a127a8ce88d3a24da8344fdd23654841b7b97b7d5e82d005ce
              prometheus.io/port: 8123
              prometheus.io/scrape: true
Status:       Running
IP:           10.90.1.173
IPs:
  IP:           10.90.1.173
Controlled By:  ReplicaSet/prod-ingress-ingress-azure-7d5ccdfb9b
Containers:
  ingress-azure:
    Container ID:   containerd://059bb0ee087c35de9b282c42b4e78351b952339c3ce94a3bf7be5af8ee013bdf
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.0-rc1
    Image ID:       mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:0bd24c33e79b60ef2ad0012bc8c636c73823cc1194b8489aa103f49b5ca1f4e9
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 10 Nov 2021 11:04:55 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      prod-ingress-cm-ingress-azure  ConfigMap  Optional: false
    Environment:
      AZURE_CLOUD_PROVIDER_LOCATION:  /etc/appgw/azure.json
      AGIC_POD_NAME:                  prod-ingress-ingress-azure-7d5ccdfb9b-9zlbc (v1:metadata.name)
      AGIC_POD_NAMESPACE:             ingress (v1:metadata.namespace)
      AZURE_AUTH_LOCATION:            /etc/Azure/Networking-AppGW/auth/armAuth.json
    Mounts:
      /etc/Azure/Networking-AppGW/auth from networking-appgw-k8s-azure-service-principal-mount (ro)
      /etc/appgw/ from azure (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4kcjg (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  azure:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/
    HostPathType:  Directory
  networking-appgw-k8s-azure-service-principal-mount:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  networking-appgw-k8s-azure-service-principal
    Optional:    false
  kube-api-access-4kcjg:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                     Age                 From                       Message
  ----     ------                     ----                ----                       -------
  Warning  FailedApplyingAppGwConfig  57m (x2 over 70m)   azure/application-gateway  network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayPathsWithinUrlPathMapMustBeUnique" Message="Paths specified in a UrlPathMap must be unique. Path /user-subscriptions/* is already specified in another PathRule in UrlPathMap /subscriptions/9defd5ce-87f9-40ce-bfac-19f5217afc5b/resourceGroups/MEE-PROD-RG/providers/Microsoft.Network/applicationGateways/MEE-PROD-AGW/urlPathMaps/mee-services-routing." Details=[]
  Warning  FailedApplyingAppGwConfig  56m                 azure/application-gateway  network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayPathsWithinUrlPathMapMustBeUnique" Message="Paths specified in a UrlPathMap must be unique. Path /marsupial/* is already specified in another PathRule in UrlPathMap /subscriptions/9defd5ce-87f9-40ce-bfac-19f5217afc5b/resourceGroups/MEE-PROD-RG/providers/Microsoft.Network/applicationGateways/MEE-PROD-AGW/urlPathMaps/mee-services-routing." Details=[]
  Warning  FailedApplyingAppGwConfig  54m (x11 over 58m)  azure/application-gateway  network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayPathsWithinUrlPathMapMustBeUnique" Message="Paths specified in a UrlPathMap must be unique. Path /external-class-management/* is already specified in another PathRule in UrlPathMap /subscriptions/9defd5ce-87f9-40ce-bfac-19f5217afc5b/resourceGroups/MEE-PROD-RG/providers/Microsoft.Network/applicationGateways/MEE-PROD-AGW/urlPathMaps/mee-services-routing." Details=[]

Thanks

[jdewisscher]

Any updates on this one ? I ran into the same issue and didn't find any way to move forward.

[krajeshrao]

anyone faced the same issue.... after updating AGIC to 1.5.2 ?