Issue with Reusing Pre-existing NLB Created via Terraform in Kubernetes Service

[wbar]

Description:

I am encountering an issue when trying to integrate a Kubernetes Service with a pre-existing Network Load Balancer (NLB) that was created via Terraform. Despite correctly tagging the NLB and configuring the Service with the necessary annotations, I receive an error indicating a conflict due to the NLB having "the same name but with different settings". This seems related to the management of Security Groups by the AWS Load Balancer Controller, which has been introduced in a recent feature update for Network Load Balancers.

Environment:

• Kubernetes version: 1.28
• AWS Load Balancer Controller version: 2.6.1, 2.7.1
• Cloud provider or hardware configuration: AWS

Steps to Reproduce:

1. Create an NLB using Terraform with specific tags:

   ingress.k8s.aws/stack	XXXXXXXX
   elbv2.k8s.aws/cluster
   ingress.k8s.aws/resource
   

2. Configure a Kubernetes Service with annotations to use the pre-created NLB and specify the load balancer settings,

    service.beta.kubernetes.io/aws-load-balancer-alpn-policy: HTTP2Preferred
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-ip-address-type: ipv4
    service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: 'false'
    service.beta.kubernetes.io/aws-load-balancer-name: XXXXXXX
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: '*'
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:ZZZZZZZ
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: SOME_ATTRS 
    service.beta.kubernetes.io/aws-load-balancer-type: external
   

3. Attempt to deploy the Service in a new environment.

Expected Result: The Kubernetes Service should successfully associate with the pre-created NLB without any conflicts regarding the load balancer's name or settings.

Actual Result: Received an error message:

A load balancer with the same name 'XXXXXXX' exists, but with different settings.

This suggests an issue with how the AWS Load Balancer Controller handles existing NLBs, particularly regarding Security Groups settings.

Additional Information:

• The error persists even after setting service.beta.kubernetes.io/aws-load-balancer-security-groups: "" in an attempt to bypass automatic Security Group management.
• Here is a snippet from the AWS Load Balancer Controller logs indicating the attempted settings:

  // resources["AWS::ElasticLoadBalancingV2::LoadBalancer"]
  {
      "LoadBalancer": {
          "spec": {
              "name": "XXXXXXX",
              "type": "network",
              "scheme": "internet-facing",
              "ipAddressType": "ipv4",
              "subnetMapping": [
                  {
                      "subnetID": "subnet-aaaaaaaaa"
                  },
                  {
                      "subnetID": "subnet-bbbbbbb"
                  },
                  {
                      "subnetID": "subnet-ggggggg"
                  }
              ],
              "securityGroups": [
                  {
                      "$ref": "#/resources/AWS::EC2::SecurityGroup/ManagedLBSecurityGroup/status/groupID"
                  },
                  "sg-TTTTTTTTTTTT"
              ]
          }
      }
  }