Ingress config with two hostnames pointing to one service creates AppGateway update Loop

[gitflo1]

trafficstars

Describe the bug We added an Ingress.yaml containing two hostnames. Both of the hostnames were pointing to one service as backend. The service contains one health probe:

To Reproduce We got three services running in the namespace myapp-ns. Additionally we applied one ingress.yaml, containing two hostnames (mydns-1.de and mydns-2.de). Both host configurations pointed to the same services in the backend. You can check the applied ingress config ingress-faulty:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-faulty
  namespace: myapp-ns
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    cert-manager.io/cluster-issuer: letsencrypt
    cert-manager.io/acme-challenge-type: http01
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
    - hosts:
        - mydns-1.de
        - mydns-2.de
      secretName: tls-secret
  rules:
    - host: mydns-1.de
      http:
        paths:
          - path: /api/*
            pathType: Prefix  
            backend:
              service:
                name: myapp-api
                port: 
                  number: 80        
          - path: /email/v1/mail-bounce-handler/*
            pathType: Prefix  
            backend:
              service:
                name: myapp-trigger-mail
                port: 
                  number: 80
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myapp-frontend
                port: 
                  number: 80
    - host: mydns-2.de
      http:
        paths:
          - path: /api/*
            pathType: Prefix  
            backend:
              service:
                name: myapp-api
                port: 
                  number: 80        
          - path: /email/v1/mail-bounce-handler/*
            pathType: Prefix  
            backend:
              service:
                name: myapp-trigger-mail
                port: 
                  number: 80
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myapp-frontend
                port: 
                  number: 80

This resulted in the AGIC chart to update the attached application gateway about every 1.3 min in a loop. I checked the ingress-controller logs in verbosity level 5 to get a clue about what's going on. It showed a diff in the health-probes! Due to both hostnames pointing to the same service, the service health-probe was conflicting. It tried to create a health-probe for mydns-1.de/health and in the next loop it overwrites it with mydns-2/health and so on... The logs further down below are underlining this issue.

The behavior was fixed as soon as one hostname was removed from the Ingress.yaml. See the attached ingress-healthy:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-healthy
  namespace: myapp-ns
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    cert-manager.io/cluster-issuer: letsencrypt
    cert-manager.io/acme-challenge-type: http01
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
    - hosts:
        - mydns-1.de
      secretName: tls-secret
  rules:
    - host: mydns-1.de
      http:
        paths:
          - path: /api/*
            pathType: Prefix  
            backend:
              service:
                name: myapp-api
                port: 
                  number: 80        
          - path: /email/v1/mail-bounce-handler/*
            pathType: Prefix  
            backend:
              service:
                name: myapp-trigger-mail
                port: 
                  number: 80
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myapp-frontend
                port: 
                  number: 80

Ingress Controller details

• Output of `kubectl logs | String-Select "BEGIN":

I0713 08:05:38.840254       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:07:01.603698       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:08:24.034032       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:09:46.597136       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:11:09.208537       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:12:31.865773       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:13:54.368128       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:15:17.269083       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:16:40.972178       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:18:03.271199       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:19:25.835534       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:20:48.403136       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:22:10.757963       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:23:33.410470       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:25:06.577100       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:26:29.307317       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:27:52.061071       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:29:14.451397       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:30:36.865233       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:31:59.510051       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:33:21.963376       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:34:44.423503       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:36:06.862373       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:37:29.911862       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:38:52.431646       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:40:15.044328       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:41:47.850160       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:43:10.284393       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:44:33.582956       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 08:45:56.452062       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
-> INFO: Here the ingress-healthy Ingress.yaml config was applied with one hostname only!
I0713 09:03:34.487514       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0713 09:04:57.486369       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment

• Any Azure support tickets associated with this issue: TrackingID#2107060050001944

It was really hard to debug, as there was no ERROR message within the logs. Not even verbosity level 5 showed an ERROR or something related to the health probes.

Nevertheless I think this is not the desired behavior and should be fixed.

[quinnj]

Bump; I'm also seeing this issue. Any other work-arounds available?