SSL termination for TCP loadbalancing

[RichardoC]

Is your feature request related to a problem? Please describe. Could this ingress controller do SSL/TLS termination when doing TCP load balancing?

Describe the solution you'd like The ability to do TLS termination for a TCP load balanced service

Describe alternatives you've considered

Additional context

Aha! Link: https://nginx.aha.io/features/IC-108

[Rulox]

Hey @RichardoC

This is a valid use case. We are currently designing the TCP LB with our CRDs, and this will include SSL termination. This feature will come soon, we're planning to add it for our next release.

In the meantime, would this work for you? You could do TLS termination by having the crt/key mounted in the Ingress Controller pod and extending that example.

[RichardoC]

That's excellent to hear! Look forward to that being released.

It was seeing that functionality which gave me the idea for using this ingress controller for both SSL termination and TCP loadbalancing. The use case I was thinking of will require the ingress controller to do both unfortunately so I can't use it yet. Thanks though!

[RichardoC]

Any updates? We'd love to take advantage of this functionality

[Rulox]

Hi @RichardoC sorry for the delay on the answer!

We've implemented TCP/UDP load balancing. Unfortunately, TLS termination for those is still on our pending list of tasks and I can't really give you an ETA right now :(

In the meantime, I was thinking that maybe TLS Passthrough can solve your problem?

[RichardoC]

Thank you for the reply!

Unfortunately that doesn't help our use case, we were hoping to offload the TLS termination to the ingress controller rather than do it in the application and still get the load balancing goodness of the controller.

[Rulox]

I see, that makes sense @RichardoC

A 'non-ideal' workaround in the meantime could be to have a different transport-server template that includes your SSL certificates (and you can mount the template and the certificates using a volume in the KIC pod). The template can be set with a cli-argument. -transportserver-template-path.

Apart from that, I'm going to follow up with the team and try to figure out if we can implement this sooner.

[pleshakov]

Hi @RichardoC

I wonder if any of the following reflect your requirements?

1. TLS termination on port 443: the Ingress Controller terminates TLS connections on port 443, where each application uses its own cert/key, and then routes TCP connections to an appropriate application (service) based on the SNI of the connection (like app.example.com). For example, app1.example.com:443 -> app1 and app2.example.com:443 -> app2. Note that the Ingress Controller also handles regular HTTPs connections on port 443 as well.
2. TLS termination on port P, other then 443: the Ingress Controller terminates TLS connections on that port, where each application uses its own cert/key, and then routes TCP connections to an appropriate application (service) based on the SNI of the connection (like app.example.com). For example, app1.example.com:P -> app1 and app2.example.com:P -> app2
3. TLS termination on multiple ports, where each application owns a dedicated port: the Ingress Controller terminates TLS connections on each port, where each application uses its own cert/key, and routes connections to appropriate application (service) based on that incoming port. For example, app1.example.com:P1 -> app1 and app2.example.com:P2 -> app2. However, the following is possible as well: random.example.com:P1 -> app1 and random.example.com:P2 -> app2 (any TLS connection regardless of the SNI on a port will be routed to the application that corresponds to that port).

[RichardoC]

After discussing with my team we had the following thoughts on each option

1. This sounds like a good option, though we could see some benefit in having the possibility of having it on a different port for networking monitoring purposes

2. This also sounds like a good option, has the advantage of being on a different port for network monitoring purposes, but does have the downside of needing another port opened on the load balancer and firewall.

3. Not good for us, as this would require a lot of ports opened for our desired setup.

One other feature we realised we'd like to use along with this is the ability to do client authentication using certificates. I just wanted to highlight this now as it may affect which option(s) you wish to pursue.

Thanks for getting back to me, and more than happy to provide more detail if you think it would help!

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[RichardoC]

Please keep this open, it would be very helpful for us

[RichardoC]

@Rulox mind removing the stale label? I'm unable to and the bot hasn't done it for me

[Rulox]

@RichardoC Sadly I don't work at NGINX anymore so I don't think I can, let me ping others in the team, @pleshakov @lucacome

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[RichardoC]

@lucacome mind removing the stale label?

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[MAXxATTAXx]

This would be a very useful feature for me too. Is this still in the roadmap?

[brianehlert]

It is. It is a matter of when we can fit the work in.

[xtianus79]

When I do this? Isn't this ssl termincation on tcp loadbalancing? This works well enough. what I want to do is terminate my mqtt client to the broker/ngress server and deliver tcp the rest of the way. This will not do that. It will open up 8883 for usage of the cert but it will fail because a self signed cert is downstream. Is this what this question is about. Using TCP as the reference rather than the layer 7 http abstraction?

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx --create-namespace --namespace $NAMESPACE --set controller.service.annotations.service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path=/healthz,controller.service.annotations.service\.beta\.kubernetes\.io/azure-dns-label-name=$DNS_LABEL --set controller.service.loadBalancerIP=$STATIC_IP --set tcp.18083=$NAMESPACE/emqx-ee:18083,tcp.1883=$NAMESPACE/emqx-ee:1883,tcp.8883=$NAMESPACE/emqx-ee:8883

When I change the above helm command to this.

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx --create-namespace --namespace $NAMESPACE --set controller.service.annotations.service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path=/healthz,controller.service.annotations.service\.beta\.kubernetes\.io/azure-dns-label-name=$DNS_LABEL --set controller.service.loadBalancerIP=$STATIC_IP --set tcp.18083=$NAMESPACE/emqx-ee:18083,tcp.1883=$NAMESPACE/emqx-ee:1883,tcp.8883=$NAMESPACE/emqx-ee:8883

Focusing on the last part: tcp.8883=NAMESPACE/emqx-ee:1883 I get the following error. Which is basically saying the termination port switch rewrite didn't occur.

Error: Client network socket disconnected before secure TLS connection was established

[brianehlert]

TCP TLS termination is available with the TransportServer resource https://docs.nginx.com/nginx-ingress-controller/configuration/transportserver-resource/ https://github.com/nginxinc/kubernetes-ingress/pull/3462