NLB TargetGroup attribute change causes downtime due to SG rule reconciliation

[bnutt]

Describe the bug Modifying the target group attributes of a loadbalancer provisioned by the LBC will result in the controller briefly revoking security group rules on the backend security group. In the time that the rules are revoked and when the rules are added back as part of reconciliation, traffic is dropped to external users. Right now this only looks to occur when using IP targets. In testing, there has been up to a minute gap between the revoke and add of the security group rules.

Steps to reproduce Provision NLB using a k8s service object. Configure the NLB with client IP preservation enabled.

apiVersion: v1
kind: Service
metadata:
  name: echoserver
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true
spec:
  selector:
    app: echoserver
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  type: LoadBalancer
  loadBalancerClass: service.k8s.aws/nlb

Allow the controller to provision the loadbalancer.

apiVersion: v1
kind: Service
metadata:
  name: echoserver
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=false
spec:
  selector:
    app: echoserver
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  type: LoadBalancer
  loadBalancerClass: service.k8s.aws/nlb

Expected outcome Client IP is disabled without downtime. If I understand the docs correctly, this may not be happening because the controller modifies the rules on what IP ranges are allowed depending whether the flag is set or not. If this is correct, is the only safe way to disable client IP preservation without downtime to provision a new loadbalancer and migrate traffic to it?

Environment

• AWS Load Balancer controller version: 2.8.1
• Kubernetes version: v1.27.13
• Using EKS (yes/no), if so version? Yes, v1.27.13. Plat version: eks.17

Additional Context:

[bnutt]

I think I have confirmed this is due to how the LB controller handles disabling client IP preservation.

1. When preserveClientIP is true, the lbc will use what is specified in loadBalancerSourceRanges and by default allow 0.0.0.0/0 for public loadbalancers.
2. When preserveClientIP is false, the lbc will use the loadbalancer subnet CIDR blocks in the security group rules, this allows only the loadbalancer ips to reach the worker node hosting the workload.
3. Since we're flipping the flag, this results in a diff the generated security group rules list. Which would be fine if the controller authorized rules before revoking them, but because it doesn't, it causes an ingress outage. Referencing https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/3639 for the revoke before authorize problem

[M00nF1sh]

@bnutt even if the controller handles the sg rules in correct order, there could still be gaps of downtimes due to the eventually consistency nature of AWS APIs, and the underlying implementation of preserve_client_ip in ELB. If you indeed need to change the preserve_client_ip, which is a big change to LBs, to avoid downtimes, i'd suggest use a new service with weighted DNS based migration.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.