AGIC doesn't work with Ingress default backend

[DreamRivulet]

Describe the bug

If we deploy Ingress with default backend and appgw.ingress.kubernetes.io/health-probe-path, the health probe will be updated randomly from the default http probe to the custom one.

To Reproduce deploy an Ingress with default backend like

• apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: appgw.ingress.kubernetes.io/health-probe-path: /health appgw.ingress.kubernetes.io/health-probe-timeout: "10"

  spec: backend: serviceName: test servicePort: 8080 rules:

  • host: xxx.xxx.xxx.com http: paths:
    • backend: serviceName: test servicePort: 8080 path: /

Every time the AGIC tries to update the appgw config, you will find the probe of this Ingress will be changed randomly.

Ingress Controller details

• Any Azure support tickets associated with this issue.

I checked the AGIC code:

and https://github.com/Azure/application-gateway-kubernetes-ingress/blob/7055fe28fb88c02b6d877e82129920269d7eff16/pkg/appgw/ingress_rules.go#L130 :

If we set the the default backend in our ingress yaml file, then there will be two backendIDs generated for each Ingress in AGIC, and the path of the default backend is set to “nil”. When generating the probe configuration, if the default backendID is populated first from the map, then the probe will be set to the default Http probe as the path is nil, if the second backendID is populated first, then it will use the custom probe.

[akshaysngupta]

Thank you for raising and providing such detail about the issue.

One easy solution to deal with this problem is fix the order of execution by sorting the ingress by namespace/name.

[torbendury]

It seems I'm also facing this bug. It can be prevented by removing the defaultBackend block from the Ingress resource. After removal, the AGIC works fine and propagates the correct backends to the correct probes.

It might be worth a note that this has a burst-radius over the whole cluster. Restarting service A (which also has an Ingress, but without defaultBackend) leads to breaking the AppGW backend for service B (which has an Ingress with defaultBackend).

[jayendranarumugam]

One easy solution to deal with this problem is fix the order of execution by sorting the ingress by namespace/name.

@akshaysngupta , Could you explain a bit about what do you mean by sorting the execution of ingress? Is there any config we need to do for achieving it ?

Looks like I'm hitting same sort of issue for which i opened https://github.com/Azure/application-gateway-kubernetes-ingress/issues/1411

when i tried with @torbendury approach on removing the defaultBackend from ingress the issue is not happening=> meaning custom probes are creating properly for the backendsettings , BUT my rule level backendtarget and backendsettings are reverting to defaults like in the below screenshot. So I'm not sure whether its a fix for mycase

[jayendranarumugam]

Whooo after a huge struggle with ms support team by going back and forth with the product team. I finally got the update

This issue is tracked internally and we plan to address this in the coming months. As a workaround, you can modify the ingress to have path “/” in the ingress pointing to the service that was used in the “defaultBackend” and omit the default backend.

Workaround will continue to work as expected after the issue is fixed. Both defaultBackend and path rule /* can used with consideration that path rule /* will be prioritized if both are specified. So, Customer doesn't need to modify their rules to use defaultBackend later.

e.g

From

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    appgw.ingress.kubernetes.io/appgw-ssl-certificate: cert-04212022
    appgw.ingress.kubernetes.io/backend-path-prefix: /
    appgw.ingress.kubernetes.io/backend-protocol: http
    appgw.ingress.kubernetes.io/health-probe-hostname: http://testapi-api.demo.com
    appgw.ingress.kubernetes.io/health-probe-interval: "45"
    appgw.ingress.kubernetes.io/health-probe-path: /apihealthcheck
    appgw.ingress.kubernetes.io/health-probe-port: "5000"
    appgw.ingress.kubernetes.io/request-timeout: "90"
    appgw.ingress.kubernetes.io/ssl-redirect: "false"
    kubernetes.io/ingress.class: azure/application-gateway
    meta.helm.sh/release-name: helm-release1
    meta.helm.sh/release-namespace: test-namespace
spec:
 defaultBackend:
    service:
      name: my-service
      port:
        number: 443
  rules:
    - host: http://testapi-api.demo.com
      http:
        paths:
        - backend:
            service:
              name: my-service
              port:
                number: 443
          path: /api
          pathType: Prefix

To:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    appgw.ingress.kubernetes.io/appgw-ssl-certificate: cert-04212022
    appgw.ingress.kubernetes.io/backend-path-prefix: /
    appgw.ingress.kubernetes.io/backend-protocol: http
    appgw.ingress.kubernetes.io/health-probe-hostname: http://testapi-api.demo.com
    appgw.ingress.kubernetes.io/health-probe-interval: "45"
    appgw.ingress.kubernetes.io/health-probe-path: /apihealthcheck
    appgw.ingress.kubernetes.io/health-probe-port: "5000"
    appgw.ingress.kubernetes.io/request-timeout: "90"
    appgw.ingress.kubernetes.io/ssl-redirect: "false"
    kubernetes.io/ingress.class: azure/application-gateway
    meta.helm.sh/release-name: helm-release1
    meta.helm.sh/release-namespace: test-namespace
spec:
 defaultBackend:
    service:
      name: my-service
      port:
        number: 443
  rules:
    - host: http://testapi-api.demo.com
      http:
        paths:
        - backend:
            service:
              name: my-service
              port:
                number: 443
          path: /
          pathType: Prefix
        - backend:
            service:
              name: my-service
              port:
                number: 443
          path: /api
          pathType: Prefix

I don't know what they meant by coming months :) . Hope this helps someone and saves their weeks/months like me

[raypettersen]

@jayendranarumugam thank you so much for this! Had the exact thing happen to me, ref #1472. Spent 2 days knocking my head against the wall.

Question now is, what happened to "coming months"? 😠

[SujithJulakanti2002]

@jayendranarumugam , can u please guide me how to remove default backend settings in AGIC

[Abbott-fu]

Hi akshaysngupta, I met the same issue with AGIC 1.5.3. This bug is still not fixed in AGIC 1.5.3. After removing "default backend" in ingress , It works for me. Please let me know when the bug will be fixed.

[Abbott-fu]

@jayendranarumugam , can u please guide me how to remove default backend settings in AGIC delete default backend in ingress spec:

defaultBackend: service: name: my-service port: number: 443

rules: - host: http://testapi-api.demo.com http: paths: - backend: service: name: my-service port: number: 443 path: / pathType: Prefix - backend: service: name: my-service port: number: 443 path: /api pathType: Prefix