Backend pools don't get updated cluster IP addresses

[baldswede]

Describe the bug I have created an AKS cluster and Application Gateway with the AGIC addon according to this tutorial.

It works out of the box, but after updating the image using kubectl set image, the service gets a new cluster IP address and the corresponding backend pool in the AG is not updated, causing the web application to break (502 Bad Gateway). Only after manually updating the target IP address in the AG's backend pool is the application reachable again.

To Reproduce Steps to reproduce the behavior:

• Create an AKS cluster, Application Gateway and enable the AGIC addon according to the MS Docs tutorial
• Browse to http://[AG public IP address], verify that the page renders
• Deploy a new version of your image to the registry and pull it in AKS using kubectl set image (rolling update)
• Browse to http://[AG public IP address] which shows 502 Bad Gateway

Ingress Controller details describe logs

Looking at the last line from the describe:

Warning FailedApplyingAppGwConfig 49m azure/application-gateway network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidResourceReference" Message="Resource /subscriptions/94ecc22d-eeaa-4d04-8218-0d4499508b83/resourceGroups/MC_MyResourceGroup_testappCluster_westeurope/providers/Microsoft.Network/applicationGateways/testappClusterApplicationGateway/backendAddressPools/pool-default-testapp-web-5002-bp-5000 referenced by resource /subscriptions/94ecc22d-eeaa-4d04-8218-0d4499508b83/resourceGroups/MC_MyResourceGroup_testappCluster_westeurope/providers/Microsoft.Network/applicationGateways/testappClusterApplicationGateway/requestRoutingRules/rr-e1903c8aa3446b7b3207aec6d6ecba8a was not found. Please make sure that the referenced resource exists, and that both resources are in the same region." Details=[]

Could it be that the ingress controller can't find or doesn't have permissions to update the AG somehow?

[aries1980]

I ran into the same bug.

In my case, the backend pool points to the pod IPs rather than the cluster IP of the service, which would be acceptable in my case, but when the pod's IP is changed, the IP in the backend pool doesn't get updated. The Ingress resource itself points to the correct backend (verified with kubectl describe ingress ingressname), but it doesn't match the with the value in the backend pool.

[gangefors]

This seems very similar to the issue I encountered. The difference is that for me this issue happens directly after deploying my application with helm, i.e. the first time the addon tries to configure the app gateway.

What I can see from the logs the template that is generated is missing the backend pool that is referenced in the routing rule.

-- App Gwy config --        "backendAddressPools": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/<redacted>/resourceGroups/myprefix-app-rg/providers/Microsoft.Network/applicationGateways/myprefix-appgw/backendAddressPools/defaultaddresspool",
-- App Gwy config --                "name": "defaultaddresspool",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "backendAddresses": []
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],

The following requestRoutingRules reference pool-jfdev-app-platform-8082-bp-8082 that doesn't exist in the above backendAddressPools list.

-- App Gwy config --        "requestRoutingRules": [
-- App Gwy config --            {
-- App Gwy config --                "id": "/subscriptions/<redacted>/resourceGroups/myprefix-app-rg/providers/Microsoft.Network/applicationGateways/myprefix-appgw/requestRoutingRules/rr-e1903c8aa3446b7b3207aec6d6ecba8a",
-- App Gwy config --                "name": "rr-e1903c8aa3446b7b3207aec6d6ecba8a",
-- App Gwy config --                "properties": {
-- App Gwy config --                    "backendAddressPool": {
-- App Gwy config --                        "id": "/subscriptions/<redacted>/resourceGroups/myprefix-app-rg/providers/Microsoft.Network/applicationGateways/myprefix-appgw/backendAddressPools/pool-jfdev-app-platform-8082-bp-8082"
-- App Gwy config --                    },
-- App Gwy config --                    "backendHttpSettings": {
-- App Gwy config --                        "id": "/subscriptions/<redacted>/resourceGroups/myprefix-app-rg/providers/Microsoft.Network/applicationGateways/myprefix-appgw/backendHttpSettingsCollection/defaulthttpsetting"
-- App Gwy config --                    },
-- App Gwy config --                    "httpListener": {
-- App Gwy config --                        "id": "/subscriptions/<redacted>/resourceGroups/myprefix-app-rg/providers/Microsoft.Network/applicationGateways/myprefix-appgw/httpListeners/fl-e1903c8aa3446b7b3207aec6d6ecba8a"
-- App Gwy config --                    },
-- App Gwy config --                    "priority": 19500,
-- App Gwy config --                    "ruleType": "Basic"
-- App Gwy config --                }
-- App Gwy config --            }
-- App Gwy config --        ],

The ingress deployed is described as

Name:             jfrog-platform
Labels:           app=artifactory
                  app.kubernetes.io/managed-by=Helm
                  chart=artifactory-107.46.11
                  heritage=Helm
                  release=jfrog-platform
Namespace:        jfdev
Address:          <redacted>
Ingress Class:    <none>
Default backend:  jfrog-platform:8082 (10.244.1.4:8082,10.244.2.4:8082,10.244.3.5:8082)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *           *     jfrog-platform:8082 (10.244.1.4:8082,10.244.2.4:8082,10.244.3.5:8082)
Annotations:  appgw.ingress.kubernetes.io/health-probe-timeout: 5
              kubernetes.io/ingress.class: azure/application-gateway
              meta.helm.sh/release-name: jfrog-platform
              meta.helm.sh/release-namespace: jfdev
Events:       <none>

And the service is described as

Name:              jfrog-platform
Namespace:         jfdev
Labels:            app=artifactory
                   app.kubernetes.io/managed-by=Helm
                   chart=artifactory-107.46.11
                   component=artifactory
                   heritage=Helm
                   release=jfrog-platform
Annotations:       meta.helm.sh/release-name: jfrog-platform
                   meta.helm.sh/release-namespace: jfdev
Selector:          app=artifactory,component=artifactory,release=jfrog-platform
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.0.100.178
IPs:               10.0.100.178
Port:              http-router  8082/TCP
TargetPort:        8082/TCP
Endpoints:         10.244.1.4:8082,10.244.2.4:8082,10.244.3.5:8082
Port:              http-artifactory  8081/TCP
TargetPort:        8081/TCP
Endpoints:         10.244.1.4:8081,10.244.2.4:8081,10.244.3.5:8081
Session Affinity:  None
Events:            <none>

[tvenegas10]

Any updates on this? I'm facing a very similar issue 2 years later