AKS with AGIC - firewall egress configuration issue

[karol-pieciukiewicz]

trafficstars

Describe the bug Pods for AAD pod identity and AGIC cannot start.

To Reproduce Create AKS cluster with:

• AGIC from standalone instance Application Gateway
• UDR for redirect egress traffic to Azure Firewall
• AAD Pod Identity
• Own vnet and subnet

Ingress Controller details Issues from PODS: Trace[1106410694]: [10.031957691s] [10.031957691s] END W0704 09:48:30.515131 1 reflector.go:324] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v1.AzurePodIdentityException: Get "https://10.0.0.1:443/apis/aadpodidentity.k8s.io/v1/azurepodidentityexceptions?limit=500&resourceVersion=0": EOF Trace[1106410694]: ---"Objects listed" error:Get "https://10.0.0.1:443/apis/aadpodidentity.k8s.io/v1/azurepodidentityexceptions?limit=500&resourceVersion=0": EOF 10031ms (09:48:30.515) E0704 09:48:30.515638 1 reflector.go:138] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.AzurePodIdentityException: failed to list *v1.AzurePodIdentityException: Get "https://10.0.0.1:443/apis/aadpodidentity.k8s.io/v1/azurepodidentityexceptions?limit=500&resourceVersion=0": EOF I0704 09:48:30.515222 1 trace.go:205] Trace[1106410694]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167 (04-Jul-2022 09:48:20.483) (total time: 10031ms): I0704 09:48:30.320360 1 trace.go:205] Trace[208240456]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:167 (04-Jul-2022 09:48:20.282) (total time: 10037ms): Trace[208240456]: ---"Objects listed" error:Get "https://10.0.0.1:443/apis/aadpodidentity.k8s.io/v1/azureassignedidentities?labelSelector=nodename%3Daks-default-25082104-vmss000000&limit=500&resourceVersion=0": EOF 10036ms (09:48:30.319

W0704 09:56:27.614244 1 supported_apiversion.go:55] Failed to get server version of the cluster: an error on the server ("") has prevented the request from succeeding

I have two UDR:

1. in the AKS vnet
2. in the APP GW vnet

Azure Firewall configuration

Configuration of the Azure Firewall based on: https://docs.microsoft.com/en-us/azure/aks/limit-egress-traffic#required-outbound-network-rules-and-fqdns-for-aks-clusters

When I make open all IPs in 443 port, then Pods are starting without issue

Maybe you faced similar issue?