support for AWS Network Load Balancer attaching sec groups in upcoming version

[emad-ramlawi]

support the recent change of Network Load Balancer now supports security groups:

https://aws.amazon.com/about-aws/whats-new/2023/08/network-load-balancer-supports-security-groups/#:~:text=Network%20Load%20Balancers%20(NLB)%20now,centrally%20enforce%20access%20control%20policies

Describe the solution you'd like

nlb.ingress.kubernetes.io/security-groups: sg-xxxx, nameOfSg1, nameOfSg2

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ClaytonOlleyNutrien]

this would be amazing, and save me from having to migrate to AWS LB controller!

[kalvkusk]

Same here, as for now it blocks using internal NLB and Global Accelerator in AWS

[kkarczewski]

Same here. Would be great to be able to use sg with ingress-nginx.

[hussainbani]

Do we know when this can happen? its now blocking to use nlb with ingress controller

[cr-actico]

It will be a really very valuable new feature to be able to use security groups on NLB's with ingress controller. We can prevent port probes with that very easy.

[IvanStanchev07]

This feature would be great to have! It would help many use cases.

[gregory-lecomte]

Same. We really need to use NLB sg to have a better security control

[ntpetrov]

There is a workaround that you can have the aws-load-balancer-controller v2.6.1 installed then you can install the ingress-nginx controller with the annotation: service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-xxx,sg-xxx,sg-xxx" Which will spoon up an NLB with the SGs attached.

[rnuzzo]

Up! Waiting for this feature so much.

[longwuyuan]

hello everyone here.

• can anyone confirm that request is for the ingress-nginx project to add that annotation in the static yaml manifest for AWS

[sorind-broadsign]

Any estimation when this feature will be available ?

[tao12345666333]

We can discuss this at the community meeting next week.

[albertoal]

In addition to the annotation shared by @ntpetrov , we also had to add the annotation below for it to work with ingress-nginx

service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: "true"

[longwuyuan]

Interesting observation

On 4/18/24 02:43, Guido Dobboletta wrote:
>
> Where did you add the annotation @albertoal ? I added it to the ingress-nginx-controller load balancer service but it didn't work. Is that supposed to go in the nginx-ingress or the AWS LBC controller ?
>

@albertoal @emad-ramlawi if there is solid proof of this annotation being a requirement, then maybe we can consider a PR to change hte project's default annotations list. The problem is this confirmation as we are not sure if it is 100% known to be a requirement on EKS

[adiii717]

I attempted to configure the service with the specified settings, but it seems that NLB is still being launched without a security group using Traefik.

service:
  spec:
    externalTrafficPolicy: Local
  annotations: 
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true
    service.beta.kubernetes.io/aws-load-balancer-attributes: deletion_protection.enabled=true
    service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-xxxxx"

Even it doesn’t respect deletion_protection.enabled=true

[longwuyuan]

Traefik related issues are not really workable in this project.

I think using the new AWS LB Controller is the optimum solution.

[adiii717]

Traefik related issues are not really workable in this project.

I think using the new AWS LB Controller is the optimum solution.

hmm, and what about this one https://github.com/kubernetes/ingress-nginx/issues/10340#issuecomment-1845546201

[kelson-pereira]

There is a workaround that you can have the aws-load-balancer-controller v2.6.1 installed then you can install the ingress-nginx controller with the annotation: service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-xxx,sg-xxx,sg-xxx" Which will spoon up an NLB with the SGs attached.

This workaround incredibly works. After much study, to install the Ingress-Nginx Controller on the EKS and thus create an NLB with Security Groups attached, two steps were necessary:

1. Install the AWS Load Balancer Controller https://docs.aws.amazon.com/eks/latest/userguide/lbc-manifest.html Each step of this procedure is necessary. Ex.: IAM policy and role, OIDC Provider created by cluster must be created on IAM Identity Provider, Install cert-manager and finally the AWS Load Balancer Controller. Note that this deployment does not create any ALB.
2. Install the Ingress-Nginx Controller https://kubernetes.github.io/ingress-nginx/deploy/#aws After downloading the deploy.yaml template, you will have to add the annotations to the Service so that an existing Security Group is attached to the NLB: service.beta.kubernetes.io/aws-load-balancer-type: "nlb" service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: "true" service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-xxxxxxxxxxxxxxxxx"

[tnimni]

There is a workaround that you can have the aws-load-balancer-controller v2.6.1 installed then you can install the ingress-nginx controller with the annotation: service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-xxx,sg-xxx,sg-xxx" Which will spoon up an NLB with the SGs attached.

This workaround incredibly works. After much study, to install the Ingress-Nginx Controller on the EKS and thus create an NLB with Security Groups attached, two steps were necessary:

1. Install the AWS Load Balancer Controller https://docs.aws.amazon.com/eks/latest/userguide/lbc-manifest.html Each step of this procedure is necessary. Ex.: IAM policy and role, OIDC Provider created by cluster must be created on IAM Identity Provider, Install cert-manager and finally the AWS Load Balancer Controller. Note that this deployment does not create any ALB.
2. Install the Ingress-Nginx Controller https://kubernetes.github.io/ingress-nginx/deploy/#aws After downloading the deploy.yaml template, you will have to add the annotations to the Service so that an existing Security Group is attached to the NLB: service.beta.kubernetes.io/aws-load-balancer-type: "nlb" service.beta.kubernetes.io/aws-load-balancer-manage-backend-security-group-rules: "true" service.beta.kubernetes.io/aws-load-balancer-security-groups: "sg-xxxxxxxxxxxxxxxxx"

@kelson-pereira This is actually the recommended AWS solution, so that ingress-nginx uses aws load balancer controller to provision the LB and not the in-tree controller see here https://aws.amazon.com/blogs/containers/exposing-kubernetes-applications-part-3-nginx-ingress-controller/