k8s-multicluster-ingress
k8s-multicluster-ingress copied to clipboard
GoogleCloudPlatform
Support layer 4 (TCP) global load balancing
Sometimes it is not appropriate to use a GCLB HTTP(S) load balancer as cluster ingress. This is common when requiring features such as...
- Custom error responses
- HTTP -> HTTPS redirects
- Support for long request headers (particularly in legacy applications)
In these case implementing HTTP(S) handling in the cluster (e.g nginx-ingress-controller) is a common solution. It would be great to take advantage of anycast IP routing and be able to balance TCP traffic across globally distributed clusters.
Thanks for filling the issue @ahume!
I understand and agree why using nginx-ingress-controller is a common solution but am not sure why do we need TCP load balancing for that?
If GCLB supported targetting IPs directly, we could use these in-cluster nginx ingresses as HTTP backends for GCLB. Anycast to Google network would remain the same in that case.
One major use case I would personally use for this is large numbers of SSL certificates being used behind a single ingress. Currently GCLB has a limit of 10 certificates. If we could do the anycast and external TCP GSLB separately from the SSL termination piece that would eliminate this capacity bottleneck. It would also allow the parts that do SSL termination to scale independently inside each cluster.
why do we need TCP load balancing for that?
I think it's that for our use cases we explicitly don't want to use GCLB. It is more complex to configure via Kubernetes Ingress and far less flexible (e.g error responses, HTTP->HTTPS redirects).
With one of our legacy applications, GCLB simply did not support the long length of HTTP request headers we were seeing. Requests were dropped and 5xx errors returned without requests ever reaching our backends. With the Network Load Balancer we can be sure that requests make it to nginx where we can handle HTTP more appropriately for our needs.
I mean that having L4 would allow us to bypass the 10-cert limit of L7 GCLB due to being able to move SSL downward into something like nginx, so I agree with @ahume . Sorry, I should have specifically mentioned "L7 GCLB".
in follow up to #203,
i believe the reason why UDP or TCP load balancing is highly desired for Federation is because beside many advantages, one of the main goals of federation is low latency (getting your server close the the endusers). latency is mostly associated with UDP protocols , point to point and none HTTP (such as: RTC, SRT, RTP, websocket , QUIC…).
+1
We are using websocket on a single cluster with multiple pods and need to use TCP with nginx-contrller to control properly the session affinity. Now we are trying to migrate to a multi-cluster architecture but encountered that we cannot migrate our websocket services to a multi-cluster because is not a TCP LB.
Reply to the below:
I think there is some confusion in terms. Regardless of the flavor of load balancing, TCP, HTTP(S), proxy, etc - load balancing in Google is called GCLB. "Layer 4" refers to the Transport Layer of the OSI model.
HTTP(S) is at Layer 7.
So when you say "this is why we don't wan't GCLB" do you mean we don't want a load balancer at all?
why do we need TCP load balancing for that?
I think it's that for our use cases we explicitly don't want to use GCLB. It is more complex to configure via Kubernetes Ingress and far less flexible (e.g error responses, HTTP->HTTPS redirects).
With one of our legacy applications, GCLB simply did not support the long length of HTTP request headers we were seeing. Requests were dropped and 5xx errors returned without requests ever reaching our backends. With the Network Load Balancer we can be sure that requests make it to nginx where we can handle HTTP more appropriately for our needs.
^^^^^
If you're looking for use cases, I am trying to migrate multi-region clusters of SOCKS and HTTP forward-proxies to Google Cloud. In an AWS environment, you'd use "latency based routing" to configure DNS records so that clients contact each region directly. In a Google Cloud environment, it seems that the only way for clients to correctly address the closest cluster is via "Global Load Balancing." If GKE multi-cluster ingress only supports HTTP/HTTPS, that makes it impossible to migrate these clusters to Google Cloud.
I have a similar story with multi-region clusters of TURN servers, but that will likely include additional challenges without a "latency based routing" equivalent.
So long as the only way to address multi-region clusters appropriately is through a "Global Load Balancer," it seems like only supporting HTTP/HTTPS will be a limitation, since there are non-HTTP services people might like to deploy to Google Cloud.
I think what @ahume is asking is whether kubemci can support other target proxies kinds. Currently it only has HTTP and HTTPS target proxies. While there also exist tcp proxies (gcloud compute target-tcp-proxies list) which are the ones created when creating these Load Balancers in the console:
My project also requires to have 1 service with HTTPS MCI and 1 service with TCP MCI (for websockets) so I second this Issue. If I get this kubemci to build from source I might be able to create a PR for this...
I see now that this is a bit more complicated, as there are not TargetTcpProxies in gce_targetproxy.go. Investigating...
@nikhiljindal do you know if there is any way to specify TCP target proxies via the k8s.io/pkg/cloudprovider/gcp interface/implementation?
+1
I need to multi-cluster TCP load balance as well. Is there any update on the progress of this issue?
