Unable to provision AGIC within ARM deployments

[mark-nexplore]

trafficstars

Describe the bug In ARM templates, it is possible to create a new AKS cluster with AGIC as an addon, but when running it through az deployment group validate the template does not accept the configuration with the error of:

{"code":"IngressAppGwAddonConfigApplicationGatewayNotFound","message":"Provisioning of resource(s) for container service aks-workforce-nx-dev in resource group rg-wf-test failed. Message: {\n  \"code\": \"IngressAppGwAddonConfigApplicationGatewayNotFound\",\n  \"message\": \"IngressApplicationGateway addon cannot find Application Gateway '/subscriptions/c6339d4b-72d9-4409-bdbf-738a79d60cbb/resourceGroups/rg-wf-test/providers/Microsoft.Network/applicationGateways/apg-workforce-nx-dev'.\"\n }. Details: "}

To Reproduce Steps to reproduce the behavior:

1. Create a new ARM template with the following configuration:

      ... other declarations like Application Gateway, Public IP. etc/
      {
            "type": "Microsoft.ContainerService/managedClusters",
            "name": "[variables('aksEntityName')]",
            "apiVersion": "2020-03-01",
            "location": "[resourceGroup().location]",
            "sku": {
                "name": "Basic",
                "tier": "[if(or(equals(parameters('env'), 'prod'), equals(parameters('env'), 'stage')), 'Paid', 'Free')]"
            },
            "dependsOn": ["[variables('vnetEntityId')]", "[variables('apgEntityId')]"],
            "properties": {
                "enableRBAC": true,
                "kubernetesVersion": "1.21.2",
                "dnsPrefix": "kcaks",
                "agentPoolProfiles": [
                    {
                        "name": "agentpool1",
                        "count": "[variables('aksEntityAgentCount')]",
                        "vmSize": "[variables('aksEntityAgentType')]",
                        "osDiskSizeGB": "[variables('aksEntityAgentDiskSize')]",
                        "osType": "Linux",
                        "vnetSubnetID": "[variables('aksEntitySubnet')]",
                        "enableAutoScaling": false,
                        "type": "VirtualMachineScaleSets",
                        "mode": "System"

                    }
                ],
                "networkProfile": {
                    "networkPlugin": "azure",
                    "serviceCidr": "10.2.0.0/16",
                    "dnsServiceIP": "10.2.0.10",
                    "dockerBridgeCidr": "172.17.0.1/16"
                },
                "addonProfiles": {
                    "omsagent": {
                        "enabled": true,
                        "config": {
                            "logAnalyticsWorkspaceResourceID": "[variables('wksEntityID')]"
                        }
                    },
                    "ingressApplicationGateway": {
                        "enabled": true,
                        "config": {
                            "applicationGatewayId": "[variables('apgEntityId')]"
                        }
                    }
                }
            },
            "identity": {
                "type": "SystemAssigned"
            }            
        },
        ...

2. Run the template with az deployment group -n test -g <some resource group> -f <template file> -p <param file>

Ingress Controller details None - unable to deploy...

[akshaysngupta]

This looks like a bug in the AKS validation logic where it is checking for app gateway existence. This validation check is happening before template deployment starts, and hence app gateway doesn't exist.

[curious-toast]

Has this been passed along so this can be addressed? Ran into the same thing.

[thepaulmacca]

For those using Bicep, this now works if using modules. Check my comment relating to a similar issue https://github.com/Azure/AKS/issues/2245#issuecomment-920704640

[brwilkinson]

+1 on this. also commenting on https://github.com/Azure/AKS/issues/2245