Incorrect ingress definition break app-gateway ingress controller refresh

[vishnujanardhanan]

Describe the bug When one of the ingress definition is incorrect, agic breaks the backend pool refresh for all the ingress objects in the cluster

To Reproduce

I0127 13:18:55.070217 1 mutate_app_gateway.go:177] BEGIN AppGateway deployment I0127 13:18:55.291922 1 mutate_app_gateway.go:183] END AppGateway deployment E0127 13:18:55.292004 1 controller.go:141] network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayPathRuleInvalidCharacter" Message="The given path /xxx-xxx-xxx?* in the path rule /subscriptions/xxxxxxxxx/resourceGroups/dev-xxx-xxx-v2/providers/Microsoft.Network/applicationGateways/xxxxx-v2/urlPathMaps/url-e1903c8aa3446b7b3207aec6d6ecba8a/pathRules/pr-xx-xx-api-xx-xxx-api-4 contains an invalid character. The allowed characters are [A–Z, a–z, 0–9, '-', '.', '', '~', '!', '$', '(' , ')', '', '\'', '+', ',', ';', '=', ':', '@']" Details=[] E0127 13:18:55.292016 1 worker.go:62] Error processing event.network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayPathRuleInvalidCharacter" Message="The given path /xx-xx-xx? in the path rule /subscriptions/xxxxxxxx/resourceGroups/dev-xx-xxx-v2/providers/Microsoft.Network/applicationGateways/dev-xxxx-xxxx-xxxxxx-v2/urlPathMaps/url-e1903c8aa3446b7b3207aec6d6ecba8a/pathRules/pr-xxx-xxxx-api-xxx-xxx-api-4 contains an invalid character. The allowed characters are [A–Z, a–z, 0–9, '-', '.', '', '~', '!', '$', '(' , ')', '*', '\'', '+', ',', ';', '=', ':', '@']" Details=[]

Kindly help

[abinet]

We have same issue with invalid liveness/readiness probe. Configuring a single live probe to start with some other character as "/" will prevents all further updates for the whole cluster.

[mxrss2]

so we hit the same thing trying to do a RegEx pattern its fine if that will never be supported by AGIC but 1 bad annotation should not choke the controller. Ideally it should be able to at least issue requests for VALID annotations.

[bbreijer]

We also hit this issue last week. In my opinion, the input (the incorrect ingress deployment) should be ignored instead of the output/update appgw step failing.

We filed SR-2305050050001271 with Microsoft Support for this issue.

Our incorrect Ingress deployment and relevant log messages below (redacted; original are available through the SR).

2023-04-25T10:05:59.547405108Z I0425 10:05:59.547329       1 requestroutingrules.go:269] Attached default redirection /subscriptions/subId/resourceGroups/development_MC/providers/Microsoft.Network/applicationGateways/akscluster/redirectConfigurations/sslr-fl-someid to rule {Host:somehost.com IngressRuleValue:{HTTP:&HTTPIngressRuleValue{Paths:[]HTTPIngressPath{HTTPIngressPath{Path:/project/<instance>/somepath/*,Backend:IngressBackend{Resource:nil,Service:&IngressServiceBackend{Name:project-resty-foo,Port:ServiceBackendPort{Name:,Number:8080,},},},PathType:*Prefix,},},}}}

2023-04-26T00:01:13.234671568Z E0426 00:01:13.234506       1 controller.go:141] network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayPathRuleInvalidCharacter" Message="The given path /project/<instance>/subpath/* in the path rule /subscriptions/subId/resourceGroups/MC/providers/Microsoft.Network/applicationGateways/akscluster/urlPathMaps/url-randomId/pathRules/pr-project-somename-ingress-rule-0-path-0 contains an invalid character. The allowed characters are [A–Z, a–z, 0–9, '-', '.', '_', '~', '!', '$', '(' , ')', '*', '\\'', '+', ',', ';', '=', ':', '@']" Details=[]

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  namespace: projectns
  name: project-ingress-foo
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/backend-path-prefix: /
    appgw.ingress.kubernetes.io/appgw-ssl-certificate: some-cert
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  rules:
    - host: somehost.com
      http:
        paths:
          - path:   /project/<instance>/subpath/*
            pathType: Prefix
            backend:
              service:
                name: project-resty-foo
                port:
                  number: 8080

[JackStromberg]

Hello folks,

I understand and appreciate the feedback on the ask to be more flexible on handling errors in the ingress input. At this time, the behavior is expected as-is, we prevent configuration wholistically, similar to ARM definition of an Application Gateway on CRUD/PUT. While I understand collecting the feedback as-is doesn't improve the experience today and I do not have an ETA to provide on when this deployment experience may change, we will certainly be taking it under consideration during our planning sessions forward.

Jack