aws-load-balancer-controller
aws-load-balancer-controller copied to clipboard
kubernetes-sigs
Dual TCP/UDP NLB shared across multiple services
Hi there,
I would like to share a single NLB across two services with both UDP and TCP ports open.
For example: serviceA - Port 550 UDP serivceB - Port 8899 TCP
I couldn't seem to find a way to do this unless using an application load balancer and ingress routes.
Is there a way to do this in the v2.0.0 release?
The major blocker was that the targetgroup annotation was only supported at an ingress level (not a service level) so there just seems no way to share a LB.
The service resource models closely to the AWS NLB resource, so we don't intend to support single LB across multiple service resources. See if specifying multiple service ports helps your use case.
If you are not able to specify both TCP and UDP ports on a service resource of type LoadBalancer, you can try using service of type NodePort. The only limitation currently is that the TCP and UDP ports cannot be the same value.
Thanks @kishorj, that's a pretty clear explanation. I will try the NodePort approach.
I was wondering why I can't specify the targetgroup with NLB like I can with the ALB? That would also be a way to resolve this, just specify the same targetgroup for multiple services (such as I can do with ingress).
I also would love to see something akin to alb.ingress.kubernetes.io/group.name on NLBs (presumably as something akin to service.beta.kubernetes.io/aws-load-balancer-group-name).
NodePort seems like a nonstarter with an autoscaling cluster - you have to go manually create additional target groups and pin your autoscaling group to each of them every time you add a new service (though honestly this is more of a failing of the way NLB target groups work - ideally you should only need one target group, not one target group per port all with identical instance lists)
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale
As far as I'm aware this shouldn't be stale. I'd still love to see this, at least.
@philomory
This controller currently supports NodePort service as well. If you use the NLB-IP mode annotation service.beta.kubernetes.io/aws-load-balancer-type: "nlb-ip" on services, even it's a nodePort service, the controller will manage the loadbalancer and targetGroups for you automatically.
Also, there have been proposal in upstream to allow dual TCP/UDP.
Mixed-protocol (TCP/UDP) Service is alpha in k8s 1.20, is this the feature ticket to track for support for the MixedProtocolLBService feature gate?
Or is this for multiple Services contributing to a single NLB, similar to #1707 and #1545?
@TBBle, this is the correct issue for mixed protocol support.
Once the MixedProtocolLBService feature gate is enabled, service of type LoadBalancer with mixed protocols should work fine without further changes with the following limitations -
- cannot have same port for both UDP and TCP, for example TCP port 53 and UDP port 53. This is a limitation on the AWS side.
- NLB dual-stack currently doesn't support UDP
@TBBle What, exactly, is the limitation on the AWS side that causes this? Doesn't the NLB listener protocol TCP_UDP cover the case of the same port over both TCP and UDP?
I assume you meant @kishorj with that question.
Per the docs, TCP_UDP is explicitly for the "same port in TCP and UDP" case.
To support both TCP and UDP on the same port, create a TCP_UDP listener. The target groups for a TCP_UDP listener must use the TCP_UDP protocol.
@TBBle You're absolutely right, I meant @kishorj. My apologies.
@kishorj, what's the cause of the limitation that a Load Balancer service with mixed protocols cannot use the same port for both TCP and UDP? It's definitely supported on the AWS side in the form of the TCP_UDP protocol type. But maybe I'm misunderstanding something here?
@philomory, you are correct, AWS supports TCP_UDP protocol type. In my prior response, I was referring to the current controller code without further changes handling services with TCP and UDP protocols.
As you mentioned, It is possible to utilize the TCP_UDP protocol type supported by AWS to combine matching TCP and UDP ports from the service spec as long as the target ports or node ports for TCP and UDP protocols are same. This is something that we have been considering to add in future releases.
So as of today there is no way to have a listener with TCP_UDP ?
I have try using service.beta.kubernetes.io/aws-load-balancer-backend-protocol: TCP_UDP but it does nothing. I'm using v2.1.3.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale
I don't think this should be marked as stale, this would still be a valuable feature
Sounds like this is something that'd be good to support, I imagine adding support for TCP_UDP would probably need to be in a few places, but would a good place to start be model_build_listener.go?
@Yasumoto, I've included the design details if you are interested
- model_build_listener.go is a good place to start
- also need to modify the deployer code to create TCP_UDP type listener
TCP_UDP listener support for NLB
Overview
NLB has support for both TCP and UDP listeners and the k8s LoadBalancer service with mixed protocol is deployed as NLB configuration using both type of listeners. However, due to limitation from the AWS ELBv2 side, listener ports for TCP and UDP cannot be the same. Use cases where both TCP and UDP listeners are used where we are currently not able to provide a reasonable solution -
- DNS with both TCP and UDP listeners on port 53
- HTTP/2 over TLS+TCP and HTTP/3 over QUIC on port 443
The AWS ELBv2 has a TCP_UDP type of listener that listens for both TCP and UDP protocol on the same port and this construct is useful for providing solution for mixed protocols in limited cases. This document describe the proposed solution with the limitations.
Proposal
In case the service spec has the same TCP and UDP ports specified, convert to TCP_UDP type listener during model generation if there exists two ports p1 and p2 in the service.Spec.Ports such that:
- (p1.protocol == UDP AND p2.protocol == TCP) OR (p1.protocol == TCP AND p2.protocol == UDP)
- p1.port == p2.port
- p1.targetPort == p2.targetPort if target type is IP
- p1.nodePort == p2.nodePort if target type is Instance
For each (p1, p2) pairs, create a listener of type TCP_UDP instead of separate TCP and UDP listeners.
Backwards compatibility/User facing changes
There are no issues with backwards compatibility. This feature does not require any user action.
Limitations
Since the target ports for both the TCP and UDP protocols have be the same, the nodePort for instance targets must be statically allocated.
I just submitted a PR that implements this using a similar strategy https://github.com/kubernetes-sigs/aws-load-balancer-controller/pull/2275
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
Greetings, I've been tracking this for a while and have become a bit unclear in connecting my use case with what currently is supported, and what will be supported in Kubernetes and in EKS. Apologies in advance if I'm asking this in the wrong thread.
AWS Use Case
Device connecting to an ingestion microservice running as a multi-replica deployment in EKS (K8s 1.21). The device is provisioned with a single endpoint (only one can be provisioned) to resolve and access the ingestion service pods.
The ingestion service application is configured with two listeners, one for TCP and one for UDP. These can be configured for any port (they do not have to be the same port).
Would like to front the ingestion service in EKS with a NLB for the usual reasons. Would like to define the NLB through Kubernetes (service manifest) as part of our environment automation (e.g. Terraform and Flux, etc.).
My understanding is that I cannot do this right now. I can't define a Kubernetes service with annotations that will create a single NLB in AWS that has one listener for TCP and another listener for UDP (even with the port values being different).
Further, that Mixed-protocol (TCP/UDP) Service, which was alpha in K8s 1.20, is what I would need to accomplish this.
Just wondering if my understanding is sound on this.
I think will need to wait until EKS supports a version of K8s where the mixed-protocol LB functionality is either GA or beta and enabled by default ..?
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue or PR as fresh with
/remove-lifecycle stale - Mark this issue or PR as rotten with
/lifecycle rotten - Close this issue or PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
Hi I need some help understanding why would quic need TCP with UDP as the same listener? isn't the whole point of using UDP is that it's available in most of devices?? The question is mainly for knowledge if someone can point me somewhere I would be thankful.
Assuming you're asking about https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/1608#issuecomment-937346660, the TCP support is for HTTP/1 and HTTP/2. As of July 2022 according to Wikipedia only 75% of web browsers support HTTP/3, so not also supporting HTTP/2 and HTTP/1 would cut off a significant fraction of users.
Also, as noted in the HTTP/3 spec, a client may prefer to attempt HTTP/1 or HTTP/2 first and then be redirected by an ALTSVC frame/Alt-Svc header to the HTTP/3 service. Although in this case the HTTP/3 service could be on any UDP port, having it on the standard HTTPS port means that clients that try HTTP/3 first (also valid by the HTTP/3 spec) will not need to then fall back to HTTP/2 and the be redirected to the correct port for HTTP/3, giving the best user experience for both client implementation approaches.
I expect over time that more and more HTTP clients will attempt HTTP/3 first, but since HTTP/3 is only currently supported on approximately 25% of websites (same Wikipedia article) I expect that current implementations will prefer HTTP/2 + Alternative Service to minimise time-to-first-byte in the most-common cases.
We saw the same thing in IPv6 adoption where in the early days, you'd try the IPv4 address first (since most clients didn't have global IPv6 connectivity) but over time that shifted towards "Try both and take whichever connects first" and by 2011 was widely "try IPv6 first".