Allow annotation to define default cert

[so0k]

Currently, the default cert is decided by picking the first certificate after sorting all matching certificates.

For my set up, I have wildcard certificates of several domains and ingress manifests for all of those domains.

I recently added a naked domain ingress request, and for this particular DNS record I changed the wildcard cert to have a SAN for the naked domain. It seems that there are validation errors if this certificate is not the default of the ALB (I don't quite understand yet why that's the case).

Would this annotation make sense?

[so0k]

I guess I could use zalando.org/aws-load-balancer-shared: false - but currently only one ingress uses that naked domain.. the other domains are using subdomains and seem to work fine - forcing 1 ALB to be solely dedicated seems like a waste of resources

[hjacobs]

I'm not completely sure if I understand the scenario. What do you mean with "default" certificate for the ALB? I tried to find such property in the ALB Listener Cloud Formation reference, but could not find it.

[mikkeloscar]

I recently added a naked domain ingress request, and for this particular DNS record I changed the wildcard cert to have a SAN for the naked domain. It seems that there are validation errors if this certificate is not the default of the ALB (I don't quite understand yet why that's the case).

Let's try to understand this first before we add workarounds. Can you give some more information about what your certs look like and what type of validation error you are getting?

We have found bugs with ALB/Certs/Cloudformation which have been confirmed by AWS, so if there is something like this we should understand it and reach out to them.

@hjacobs default cert is the one specified here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-elasticloadbalancingv2-listener.html#cfn-elasticloadbalancingv2-listener-certificates all the rest is under: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-elasticloadbalancingv2-listenercertificate.html

[so0k]

All my Certs are wildcard issued through ACM:

*.foo.com
*.bar.com

Everything was working fine with ingress requests like:

ingress:
  hosts:
  - a.foo.com
ingress:
  hosts:
  - b.foo.com
ingress:
  hosts:
  - a.bar.com

Yesterday, I added an ingress request for www and naked domain for a new TLD

ingress:
  hosts:
  - www.qux.com
  - qux.com

external-dns created 2 alias records for the ALB. Browsing to https://www.qux.com worked, but when browsing to https://qux.com, a certificate error was shown: Server presented certificate for *.bar.com

Upon inspecting the ALB, *.bar.com was set as the default certificate on the 443 listener.

after making sure my certificate for *.qux.com had a SAN of qux.com, and manually setting it as the default certificate for the ALB, browsing to https://qux.com worked as expected

[szuecs]

@s0k0 I agree that this should be addressed and we should also have a workaround here but for me it sounds like an ALB issue. SNI should cover this situation, because it is very common that wildcard certs also have the pure domain in SAN.

[so0k]

I agree SNI should cover this situation, I also looked into https://github.com/zalando-incubator/kube-ingress-aws-controller/issues/149 and realised my issue is different.

For now, my manual change works until something causes controller to re-sync the ALB

[szuecs]

@so0k would you mind looking into the source code and create a change?

[szuecs]

@so0k another question would be if you want to define the default cert in the annotation by hostname or by ARN. ARN would not be ambiguous, so I would prefer it.

Anyone else thinks different? @hjacobs @mikkeloscar

[so0k]

agree on using ARN, identical to how we used to provision nginx-ingresss (using Terraform DataSource to render the ARN into the cluster bootstrap scripts)

I'll have a look at it next week.

[dintel]

@so0k Did you have any progress on this or did you implement any workaround for this issue?

[szuecs]

@dintel can you add what issue do you have? We detect certificates automatically and use all of them in the https listener (up to 25 per alb), in case of more certificates we create more ALBs. We rely on the SNI implementation of ALBs to select the certificate. I guess we can easily add the cert by arn annotations, we already had that in the past. I would like to understand what the issue is you have.

[so0k]

Sorry I didn't make progress, we iterated on many different configurations mostly using Terraform and by the time we had a working solution (NLB with TLS termination in Envoy Edge nodes) - we moved on and it stayed in Terraform) - I have also since moved to another company and will most likely not have the time to work on this.

[dintel]

Sorry for a delay. In our case we have a few certificates in use and they all are assigned to ALB correctly, but default certificate is always chosen alphabetically by ingress controller. It does not matter for SNI enabled clients, but we need to configure default certificates for clients that don't support SNI. The question is how this could be fixed? Annotation? What if multiple ingresses have this annotation with different value? May be add a command line parameter with a list of ARNs of preferred default certificates?

[szuecs]

@dintel Right now you can not choose default certs. If you have different ingress with different values you should get additional ALBs provisioned. You can use zalando.org/aws-load-balancer-ssl-cert to assign a cert ARN via ingress annotation such that you can pin a certificate to an ingress, but this was before we used the SNI shared ALBs.

[dintel]

@szuecs We are using SNI and shared ALBs (public and internal) with kube-ingress-aws-controller. But we want to be able to setup default certificate that will be used for clients that do not support SNI. IMO setting this in annotation is not best way (because you might set conflicting values in different ingress objects). Therefore I thought about setting this in command line parameter.