Kubenet: Add node routes to Application Gateway subnet route table

[tiwood]

For our AKS Workloads we have two subnets with two route tables:

• AppGatewaySubnet, where we put all Application gateways.
• A AKS specific subnet, where the nodes live.

AKS recently introduced a new feature which automatically adds the Pod CIDR ranges to the preconfigured routing table (in our case the AKS routing table), like so:

10.244.0.0/24 --> 172.28.0.100
10.244.1.0/24 --> 172.28.0.101

We would love if AGIC would do the same for our AppGatewaySubnet routing table.

I've seen in the 1.2 RC changelog that you've added support for attaching the routing table of the AKS subnet to the AppGW subnet, which won't work for us as we require different routes on the subnets - default route to our Firewall appliance for example, which is not supported by AppGWv2.

[akshaysngupta]

@tiwood that's a great suggestion.

I am wondering about how would an implementation look like. One potential way could be that AGIC copies over the route table rules (name starting with "aks-nodepool") from AKS subnet to AG subnet's route table.

Suggestions are welcome.

[tiwood]

Your suggestion seems like a good way to do it:

1. Authentication (Managed Identity, Service Principal)
2. Get AKS Subnet
3. Get Subnet Route table and routes (aks-subnet-rt)
4. Get AppGW route table (if any) (appgw-rt)
5. Copy routes where the name starts withaks-nodepool- from aks-subnet-rt to appgw-rt

Considerations:

• This will also copy rules for other clusters (if you have the same route table for multiple clusters) - which should be ok (we can log a warning) Optional Only copy routes where the CIDR is "inside" the AKS Cluster POD CIDR
• We should throw if there is a route with a different name but same CIDR. (This may happen if you have multiple clusters with the same POD cidr.)
• We should use the same identity AGIC is using for AppGW configuration. (we use a service principal)
• This should be an opt-in feature (like addNodeRoutesToAppGWRouteTable: true)

[mkarebski]

We have a similar problem.

Our infrastructure uses spoke-hub topology. One of the reasons for using AGIC with Kubneet CNI was to reduce consumption of IP address space which is limited in our case.

We figured out a potential solution similar to what @tiwood suggested with Azure Automation, but it seems hacky and in case of downscaling, could result in losing connectivity (502's when hitting Application Gateway). Ofc. we can also implement custom controller in Go running in Kubernetes which continuously watches autoscaling events and adds route entries accordingly, but this requires additional engineering effort.

Apart from that, we thought about an internal load balancer in front of AKS cluster so we can provide only one IP, but we are not sure about it either. Below diagram shows the idea:

Currently, our solution is based on Application Gateway in standalone mode (without ingress controller), with configured directly in an internal load balancer backend pool.

external client —(TLS termination)—> Application Gateway —(backend pool)—> Internal Load Balancer -> Nginx Ingress Controller —(Ingress manifest)—> Pod

[Tbohunek]

Hi there, any news on this topic? 😉 I wanted to promote to use AGIC in our company only to find out it ain't supported with kubenet.

We are at exactly the same case, both subnets need to have different route tables. @tiwood according to Docs (and logic), no 2 clusters should ever share a Route table, in which case just copying routes shall be safe.

Process-wise I would say that whenever AGIC receives new backend to add, it should check whether it has a route for that and if no, copy it over. I think that's safer than watching for route table changes, but it's more operations...

[deepak-malhotra]

@akshaysngupta Is this coming out soon ? All our AKS deployments are with Kubenet with default route pointing to Az FW.

Using AGIC is not possible without an automation to auto-populate/ delete routes from AAG route table . Is this on roadmap ?

[jack4it]

It would be great to have some traction on this one as we are using app gateway and meanwhile wants to restrict the aks egress traffic via a Firewall

[mkarebski]

Hi there, I just want to know if this issue is still relevant? :)

My teammates and I developed a solution for that for our client. It's a custom-made operator in Kubernetes, which rewrites rules from one RT to another.

Does such an operator solve the problem? I believe for Kubernetes-based solution - yes, but I want to hear the community :)

Please add +1 under this post if it's relevant and it solves your problem.

[davemcdonnell]

Im not using firewall but this feature would be helpful for me to fix some asymmetric routing issues I have with AKS/AGIC and ExpressRoute.

[kamilzzz]

I'm facing the same issue with AGIC and kubenet as I would like to control cluster's egress traffic using Azure Firewall.

[OmpahDev]

Facing the same issue, need to have an egress firewall from AKS and an AGIC.

[ebc92]

My organization is facing this issue as well. We would like to use AGIC, but because we are using AzFw for egress we are currently looking into the solution described by @mkarebski where AppGw is set up and configured independently and pointing to the cluster ILB.

[extromen13]

Hi there, I just want to know if this issue is still relevant? :)

My teammates and I developed a solution for that for our client. It's a custom-made operator in Kubernetes, which rewrites rules from one RT to another.

Does such an operator solve the problem? I believe for Kubernetes-based solution - yes, but I want to hear the community :)

Please add +1 under this post if it's relevant and it solves your problem.

Hey, @mkarebski

could share your solution, please? It would help a lot Thank you very much

[kamilzzz]

In context of egress filtering via NVA with kubenet and AGIC.

Recently some improvements for Application Gateway v2 were announced and it looks like now it is supported to have 0.0.0.0/0 route pointing to NVA attached to Application Gateway's subnet, so it should be possible to let AGIC assign kubenet's route table to Application Gateway. Unless you have some other rules in kubenet route table which you wouldn't like to have on Application Gateway subnet.

https://azure.microsoft.com/en-us/updates/public-preview-private-application-gateway-v2/ https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-private-deployment?tabs=portal#introduction

[rrileyca]

Is this issue being worked on? The route table doesn't work if the AKS and App GW have different default gateways (0.0.0.0/0 routes).