routing to multiple hosts with different SSL certs ARN with NLB

[wwteo]

Describe the bug The service.beta.kubernetes.io/aws-load-balancer-ssl-cert annotation can only assign one SSL certificate from ACM. According to documentation, it should be able to assign multiple SSL certificates as it takes in a StringList https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/main/docs/guide/service/annotations.md The controller issues a warning event that assumes a string instead of a stringlist. Error syncing load balancer: failed to ensure load balancer: error updating load balancer listener: "ValidationError: Certificate ARN 'arn:aws:acm:<region>:<account_id>:certificate/<uuid>,arn:aws:acm:<region>:<account_id>:certificate/<uuid>' is not valid\n\tstatus code: 400, request id: 2091d6fd-c98c-4a3d-a1ec-db38aab498df"

Steps to reproduce Place a stringlist into service.beta.kubernetes.io/aws-load-balancer-ssl-cert Expected outcome The NLB should have multiple SSL certificates on its listeners

Environment

• AWS Load Balancer controller version 2.4.2
• Kubernetes version
• Using EKS (yes/no), if so version? yes 1.21

Additional Context:

PS: When manually assigning multiple SSL certificates in the NLB with the AWS console, the ingress works via hosts routing.

[empeekdev]

We have exactly the same issue that happened to us few minutes ago

P.S: certificates are from single region and generated by ACM ( not imported)

[empeekdev]

@wwteo was you able to solve this issue?

[wwteo]

@wwteo was you able to solve this issue?

no

[iusergii]

EKS - v1.22.9-eks-a64ea69, Controller - 2.4.2 works for me:

service.beta.kubernetes.io/aws-load-balancer-ssl-cert: >-
          arn:aws:acm:us-east-1:xxxxxxx:certificate/884f730d-4f4e-490d-bca9-31604f218240,
          arn:aws:acm:us-east-1:xxxxxxx:certificate/ea98d10e-c7e8-4ed9-bc29-8ba12a22c437,
          arn:aws:acm:us-east-1:xxxxxxx:certificate/8a5614c8-4e27-435d-b624-ff8437b3ba9b,
          arn:aws:acm:us-east-1:xxxxxxx:certificate/34261a24-5594-4319-a62f-73a6ef8a6c43,
          arn:aws:acm:us-east-1:xxxxxxx:certificate/a27aaffc-ea6d-466f-bfb8-697c1a869fe4,
          arn:aws:acm:us-east-1:xxxxxxx:certificate/156da266-d4ab-47ab-9a2b-d5358a22df3b,
          arn:aws:acm:us-east-1:xxxxxxx:certificate/23e04e3a-9ffd-4613-bb8c-41b7336812bb

[jozenstar]

@empeekdev @wwteo Have you guys tried the solution suggested by @iusergii?

[minusv]

I tried the solution suggested by @iusergii but it didn't work for me. Does anyone else have a working solution for this? Meanwhile, @iusergii Can you share the full YAML config of your solution? Could be something else that we might be missing.

[kishorj]

@minusv, @wwteo, could you share the full manifest of the service? Did you ensure the NLB got provisioned by this controller? If the LB name starts with k8s-, then it is provisioned by this controller. Also, verify if the ACM certificates are still valid and did not expire.

[minusv]

@kishorj Thanks for pointing this out. The NLB created didn't have k8s- in the prefix. I checked this document here and found that ALB-controller will only spawn an LB of type ALB when --ingress-class=alb is passed as argument. This was the case in my setup.

Any idea on how can I allow ALB-controller to spawn both nlb and alb in the cluster?

EDIT: Based on the doc I can remove --ingress-class=alb and it'll spawn LB for all the classes. But I'm using Kong Ingress Controller in my cluster and don't want an LB corresponding to ingressClassName: kong.

So my requirement is:

1. Spawn NLB and ALB via ALB-controller for ingressClassName: alb and ingressClassName: nlb only.
2. Attach multiple certs on both NLB and ALB.

[kishorj]

IngressClass is for ALB. For NLB, you will have to provision a service resource of type LoadBalancer, and the following annotations:

service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: instance

or

service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

You could also set the service spec.loadBalancerClass to service.k8s.aws/nlb on k8s 1.22 or later in-lieu of the aws-load-balancer-type type annotation.

[kishorj]

@wwteo, I'm closing the issue since it is possible to attach multiple SSL certificates to the NLB provisioned by this controller. if you still face issues, feel free to reach out.

[FalconerTC]

I'm also running into this issue, with EKS 1.21 and controller 2.4.4. If @iusergii got it to work, I assume it's fixed in EKS 1.22+

edit: Never mind, got it to work. In my case I had set

service.beta.kubernetes.io/aws-load-balancer-type: nlb

Which is not valid and it caused the legacy controller to interpret the service. Changing it to external fixed this.