aws-load-balancer-controller icon indicating copy to clipboard operation
aws-load-balancer-controller copied to clipboard

Published 20 hours ago • verified publisher icon kubernetes-sigs

Overwrite the configuration of NLB using the same NLB name

Open davidshtian opened this issue 2 years ago • 9 comments

Describe the bug The original idea it to reuse NLB for multiple services for easier management, while target group binding might be a workaround for this, there is still risk for this that if some team member using the same NLB for a new service, all the configuration of NLB will be overwrited and thus impact the business, as there may not a name verify for existing NLB.

Steps to reproduce Create 2 services with same service.beta.kubernetes.io/aws-load-balancer-name annotation but different configurations, and the second one will overwrite the first one.

Expected outcome It will check if the NLB exists or not, and it either throws an error or append the new listener to NLB.

Environment

  • AWS Load Balancer controller version: v2.*
  • Kubernetes version: *
  • Using EKS (yes/no), if so version? Yes

Additional Context: Similar issues: #1707 , #1608

davidshtian avatar Mar 02 '22 02:03 davidshtian

@davidshtian, you should be able to specify the same name for two services and end up with two separate NLB. We will verify it on our side and fix it if this is not the case. EDIT: two services cannot have the same name via annotation. The second one will fail.

kishorj avatar Mar 02 '22 23:03 kishorj

/assign

haouc avatar Mar 02 '22 23:03 haouc

@davidshtian, you should be able to specify the same name for two services and end up with two separate NLB. We will verify it on our side and fix it if this is not the case.

I've tried using v2.2+ controller and it will only create one NLB with same name annotation for two or more different services, and the last created conf will overwrite the previous ones, but not append new listener to NLB.

davidshtian avatar Mar 03 '22 02:03 davidshtian

Besides, from logs of controller pod it seems that to deploy the model, the controller first clean up the previous listener. Hope the information is helpful. Thanks

{"level":"info","ts":1646284569.4988031,"logger":"controllers.service","msg":"creating loadBalancer","stackID":"default/my-nginx","resourceID":"LoadBalancer"}
{"level":"info","ts":1646284569.9104574,"logger":"controllers.service","msg":"created loadBalancer","stackID":"default/my-nginx","resourceID":"LoadBalancer","arn":"arn:aws:elasticloadbalancing:us-east-1::loadbalancer/net/my-nginx/1839f8b3b12b8"}
{"level":"info","ts":1646284570.0128255,"logger":"controllers.service","msg":"deleting listener","arn":"arn:aws:elasticloadbalancing:us-east-1::listener/net/my-nginx/1839f8b3b12b8/3d2437a701582"}
{"level":"info","ts":1646284570.1205678,"logger":"controllers.service","msg":"deleted listener","arn":"arn:aws:elasticloadbalancing:us-east-1::listener/net/my-nginx/1839f8b3b12b8/3d2437a701582"}
{"level":"info","ts":1646284570.1206074,"logger":"controllers.service","msg":"creating listener","stackID":"default/my-nginx","resourceID":"9376"}
{"level":"info","ts":1646284570.2031991,"logger":"controllers.service","msg":"created listener","stackID":"default/my-nginx","resourceID":"9376","arn":"arn:aws:elasticloadbalancing:us-east-1::listener/net/my-nginx/1839f8b3b12b8/b24ae68ddc8f1"}
{"level":"info","ts":1646284570.2378979,"logger":"controllers.service","msg":"successfully deployed model","service":{"namespace":"default","name":"my-nginx"}}

davidshtian avatar Mar 03 '22 06:03 davidshtian

Faced same issue, unknowingly created a duplicate https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/2536

rverma-dev avatar Mar 04 '22 17:03 rverma-dev

@davidshtian @rverma-nsl The names of LoadBalancers must be unique within region. So the correct behavior should be the 2nd Service will failed to be reconciled due to the name is already token by another existing NLB.

Hasn't tested this yet, but I guess it's due to ELB's API idempotency, thus when we try create the 2nd NLB with same name, it returned the ARN of previous created one. We'll validate above and fix if that's the case (the fix might be validate the tags on the returned NLB, and issue a k8s event about NLB with same name already exists)

For customer use cases, we should assign different names per service, or let the controller auto-generate LB names by not specify the service.beta.kubernetes.io/aws-load-balancer-name annotation

/kind bug

M00nF1sh avatar Mar 09 '22 23:03 M00nF1sh

@M00nF1sh You are right, I run into the same issue and confirmed that the root cause is that CreateLoadBalancer will return the existed NLB ARN if there already have NLB with the same name.

Just running following command twice, the output will be the same, no errors in the second call:

$ aws elbv2 create-load-balancer --name whe-lb-test --type network --subnet-mappings SubnetId=subnet-0123456789 
{
    "LoadBalancers": [
        {
            "LoadBalancerArn": "arn:aws:elasticloadbalancing:ap-northeast-1:0123456789:loadbalancer/net/whe-lb-test/124880a5d69e957a",
            "DNSName": "whe-lb-test-0123456789.elb.ap-northeast-1.amazonaws.com",
            "CanonicalHostedZoneId": "Z31USIVHYNEOWT",
            "CreatedTime": "2022-04-14T09:35:20.815Z",
            "LoadBalancerName": "whe-lb-test",
            "Scheme": "internet-facing",
            "VpcId": "vpc-0123456789",
            "State": {
                "Code": "provisioning"
            },
            "Type": "network",
            "AvailabilityZones": [
                {
                    "ZoneName": "ap-northeast-1a",
                    "SubnetId": "subnet-0123456789",
                    "LoadBalancerAddresses": []
                }
            ],
            "IpAddressType": "ipv4"
        }
    ]
}

walkley avatar Apr 14 '22 09:04 walkley

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot avatar Jul 13 '22 10:07 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot avatar Aug 12 '22 10:08 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue or PR with /reopen
  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-triage-robot avatar Sep 11 '22 10:09 k8s-triage-robot

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Reopen this issue or PR with /reopen
  • Mark this issue or PR as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot avatar Sep 11 '22 10:09 k8s-ci-robot

Since the new NLB will override the existing one with the same name, the expected behaviour then should be to fail the NLB creation if the load balancer name is already used.

eslam-gomaa avatar Feb 05 '23 22:02 eslam-gomaa

@abohne: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot avatar Jun 05 '23 18:06 k8s-ci-robot

This issue should not be closed. It is still an existing bug.

abohne avatar Jun 05 '23 18:06 abohne

I'm facing this issue using v2.6 of aws-load-balancer-controller, the weird thing is that two weeks ago I tried to create 3 services attached to the same NLB using different ports in v2.5.4 of aws-load-balancer-controller and it worked. Particularly I used these annotations at this moment.

    service.beta.kubernetes.io/aws-load-balancer-name:
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type:
    service.beta.kubernetes.io/aws-load-balancer-scheme:
    service.beta.kubernetes.io/aws-load-balancer-subnets:
    service.beta.kubernetes.io/aws-load-balancer-type:

Hint 💡 Uninstall actual version with helm delete aws-load-balancer-controller -n kube-system Install the v2.5.4 that works with the helm chart v1.5.5 with

helm install aws-load-balancer-controller eks/aws-load-balancer-controller --version 1.5.5 -n kube-system --set clusterName=YOUR-CLUSTER-NAME --set serviceAccount.create=false --set serviceAccount.name=YOUR-SA-CREATED-BEFORE-FOR-THIS

4rth4S avatar Aug 23 '23 19:08 4rth4S