Feature: Allow setting of priority order rule

[chary1112004]

Describe the bug We are facing issue with priority order rule

ingress manifest

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/actions.response-403: |
      {"type":"fixed-response","fixedResponseConfig":{"contentType":"text/plain","statusCode":"403","messageBody":"403 External access to endpoint not allowed"}}
    alb.ingress.kubernetes.io/actions.root-redirect: '{"Type": "redirect", "RedirectConfig":
      { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301", "Path": "/auth"}}'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":
      { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/backend-protocol: HTTPS
    alb.ingress.kubernetes.io/group.name: alb-ingress
    alb.ingress.kubernetes.io/healthcheck-path: /health
    alb.ingress.kubernetes.io/healthcheck-port: "8443"
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTPS
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-FS-1-2-Res-2019-08
    alb.ingress.kubernetes.io/target-type: ip
    kubernetes.io/ingress.class: alb
    meta.helm.sh/release-name: application
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2022-09-19T04:10:23Z"
  finalizers:
  - group.ingress.k8s.aws/alb-ingress
  generation: 2
  labels:
    app: application
    app.kubernetes.io/managed-by: Helm
  name: application
  namespace: default
  resourceVersion: "417101858"
  uid: c7047b08-d71c-455b-a6cb-88277b14fe05
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          service:
            name: response-403
            port:
              name: use-annotation
        path: /metrics
        pathType: ImplementationSpecific
      - backend:
          service:
            name: application-service
            port:
              number: 8443
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - example.com
    secretName: example-secret

Expected outcome

• We expect in ALB rule, rule with path /metrics will have higher priority than rule with path / then while we access from https://example.com/metrics it still returns metric data that we expect it returns 403.
• We try move backend service with path / to defaultBackend however if we deploy 2 same ingresses (with different host) that include defaultBackend then it will generate below error. Then we expect in ingress support to set priority order rule

Warning  FailedBuildModel        8s (x13 over 30s)    ingress  Failed build model due to multiple ingress defined default backend:

Any solution to resolve this issue would be very much appreciated!

Environment

• AWS Load Balancer controller version: v2.5.2
• Kubernetes version: 1.25
• Using EKS (yes/no), if so version? yes

[ysam12345]

Hi @chary1112004 ,

From my understanding, if you want to configure rule priority, a way is to create two Ingresses with the same alb.ingress.kubernetes.io/group.name [1] annotaion, and configure the Ingress order with alb.ingress.kubernetes.io/group.order [2] annotation.

For example, you can set the Ingress with rule path /metrics with smaller number of group.order, so it's rule will be evaluated first.

[1] AWS Load Balancer Controller - Ingress annotations - alb.ingress.kubernetes.io/group.name https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.6/guide/ingress/annotations/#group.name

[2] AWS Load Balancer Controller - Ingress annotations - alb.ingress.kubernetes.io/group.order https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.6/guide/ingress/annotations/#group.order

[chary1112004]

Hi @ysam12345,

Thank you for your suggestion. As I know if we create new group then it means new application load balancer will be created. Does it cost for this new application load balancer (for example without throughput)?

Regards

[ysam12345]

Hi @chary1112004 ,

What I mean is to use multiple Ingresses with the same alb.ingress.kubernetes.io/group.name. In this case, controller will only create a single ALB for multiple Ingresses, the rules in multiple Ingresses will be aggregate into a single ALB, just like the document said here [1]:

IngressGroup feature enables you to group multiple Ingress resources together. 

The controller will automatically merge Ingress rules for all Ingresses within IngressGroup and support them with a single ALB. 

In addition, most annotations defined on an Ingress only apply to the paths defined by that Ingress.

So it shouldn't create a new ALB without throughput. I hope this helps.

[1] IngressGroup https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.6/guide/ingress/annotations/#ingressgroup

[chary1112004]

Hi @ysam12345, thank you. I thought the group order is for ordering between groups, not same group.

I will give a try. Thank you!

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[shraddhabang]

The priority on the rules is decided on the PathType. The prefix type take higher priority here than the implementations specific. https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/guide/ingress/spec/#ingress-specification Have you tried setting appropriate pathtype on your rules?

[chary1112004]

@shraddhabang thanks for the follow up. Since we have tried do manual step to re-order to work around currently then we did not have a chance to come back this again. I will close this one since there is already suggestion from @ysam12345

[chary1112004]

Hi @ysam12345,

We are coming back this one.

I have checked again then what you mean it could be for different hosts. For our case, we configure for same host (in this one is host: example.com) with different path.

@shraddhabang: yes, however we try to switch between prefix and implementations specific then there is error in ingress: prefix path shouldn't contain wildcards

[ysam12345]

Hi @chary1112004 ,

My reply is not specifically for the usecase of different hosts, you can define the same host with different path in different ingresses, because those Ingress Rules will be eventually transfered to ALB Listener rules.

Example 1

group.order 10 -> 20 -> 30

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-1
  annotations:
    alb.ingress.kubernetes.io/group.order: '10'
    alb.ingress.kubernetes.io/group.name: my-ingress-group
spec:
  ingressClassName: alb
  rules:
    - host: foo.bar
      http:
        paths:
          - path: /*
            pathType: ImplementationSpecific
            backend:
              service:
                name: "foo-bar"
                port:
                  number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-2
  annotations:
    alb.ingress.kubernetes.io/group.order: '20'
    alb.ingress.kubernetes.io/group.name: my-ingress-group
spec:
  ingressClassName: alb
  rules:
    - host: foo.bar
      http:
        paths:
          - path: /aaa
            pathType: ImplementationSpecific
            backend:
              service:
                name: "foo-bar"
                port:
                  number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-3
  annotations:
    alb.ingress.kubernetes.io/group.order: '30'
    alb.ingress.kubernetes.io/group.name: my-ingress-group
spec:
  ingressClassName: alb
  rules:
    - host: foo.bar
      http:
        paths:
          - path: /bbb
            pathType: ImplementationSpecific
            backend:
              service:
                name: "foo-bar"
                port:
                  number: 80

Example 2

group.order 30 -> 20 -> 10

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-1
  annotations:
    alb.ingress.kubernetes.io/group.order: '30'
    alb.ingress.kubernetes.io/group.name: my-ingress-group
spec:
  ingressClassName: alb
  rules:
    - host: foo.bar
      http:
        paths:
          - path: /*
            pathType: ImplementationSpecific
            backend:
              service:
                name: "foo-bar"
                port:
                  number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-2
  annotations:
    alb.ingress.kubernetes.io/group.order: '20'
    alb.ingress.kubernetes.io/group.name: my-ingress-group
spec:
  ingressClassName: alb
  rules:
    - host: foo.bar
      http:
        paths:
          - path: /aaa
            pathType: ImplementationSpecific
            backend:
              service:
                name: "foo-bar"
                port:
                  number: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-3
  annotations:
    alb.ingress.kubernetes.io/group.order: '10'
    alb.ingress.kubernetes.io/group.name: my-ingress-group
spec:
  ingressClassName: alb
  rules:
    - host: foo.bar
      http:
        paths:
          - path: /bbb
            pathType: ImplementationSpecific
            backend:
              service:
                name: "foo-bar"
                port:
                  number: 80

I'm not sure if this meets your requirement, also please ignore the HTTP 503 response code in the screenshot becuase I didn't define "foo-bar" in my environment.

[chary1112004]

Hi @ysam12345,

I have added multiple ingress with same host and different group order (in same group) then it works for me.

Thanks for your support!

[omerurhan]

This is also worked for host definitions. Thanks a lot.