application-gateway-kubernetes-ingress icon indicating copy to clipboard operation
application-gateway-kubernetes-ingress copied to clipboard

AG ingress ignores changes to default backend

Open shirolimit opened this issue 2 years ago • 0 comments

Describe the bug Application gateway ignores k8s events related to defaultBackend pods/endpoints. As a result, it doesn't update the backend pool target address on pod recreation.

To Reproduce

  1. Enable Azure ingress in the cluster
  2. Apply the following yaml:
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: my-nginx
      labels:
        app.kubernetes.io/name: "my-nginx"
    spec:
      replicas: 1
      selector:
        matchLabels:
          app.kubernetes.io/name: "my-nginx"
      template:
        metadata:
          labels:
            app.kubernetes.io/name: "my-nginx"
        spec:
          containers:
            - name: nginx
              image: nginx:1.20
              ports:
                - containerPort: 80
                  name: http
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: default-nginx
      labels:
        app.kubernetes.io/name: "default-nginx"
    spec:
      replicas: 1
      selector:
        matchLabels:
          app.kubernetes.io/name: "default-nginx"
      template:
        metadata:
          labels:
            app.kubernetes.io/name: "default-nginx"
        spec:
          containers:
            - name: nginx
              image: nginx:1.21
              ports:
                - containerPort: 80
                  name: http
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: default-service
    spec:
      ports:
        - name: http
          protocol: TCP
          port: 80
          targetPort: http
      selector:
        app.kubernetes.io/name: "default-nginx"
      type: ClusterIP
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: my-service
    spec:
      ports:
        - name: http
          protocol: TCP
          port: 8080
          targetPort: http
      selector:
        app.kubernetes.io/name: "my-nginx"
      type: ClusterIP
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: test-ingress
      annotations:
        kubernetes.io/ingress.class: azure/application-gateway
        appgw.ingress.kubernetes.io/backend-path-prefix: "/"
    spec:
      defaultBackend:
        service:
          name: default-service
          port:
            number: 80
      rules:
        - http:
            paths:
              - path: /test-my-ingress/*
                pathType: Prefix
                backend:
                  service:
                    name: my-service
                    port:
                      number: 8080
    
    
  3. Check that both services work
  4. Restart default-nginx: kubectl rollout restart deployment/default-nginx
  5. Wait 30 seconds
  6. Make sure that default backend is not accessible and AG returns 502/504

Ingress Controller details

How to fix

Add

for _, ingress := range c.ListHTTPIngresses() {
	defaultBackend := ingress.Spec.DefaultBackend
	if defaultBackend != nil && (defaultBackend.Service.Name == service.Name) {
		return true
	}

	// ...
}

to https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/pkg/k8scontext/context.go#L850-L866

At the moment it only cares about rules.

shirolimit avatar Dec 22 '21 11:12 shirolimit