RFC: Client side load balancer

[mattrobenolt]

Within gRPC, one big feature is service discovery via xDS and other client side load balancing features. This is actually one of the biggest selling points imo for running gRPC internally for communication between services without needing to introduce another external proxy such as Envoy to facilitate this.

Now, I understand with connect-go, we are free to use a gRPC client and maintain these features, but would be nice to see a more official connect-go implementation.

Granted, one nice part about connect-go is the fact that you can just use a normal http.Client or specifically, an HTTPClient interface which the default http.Client implements.

What I'd like to propose is some general purpose, likely an http.RoundTripper that implements some basic features of the gRPC load balancer. I believe the gRPC implementation is too... much, but having basic connection pooling and round robin behavior per-IP address opens up the door to basic service discovery and getting the max value out of it.

For example, between internal services, let's say in Kubernetes, it's common to reference an A record that represents the service itself. This A record contains multiple IP addresses. A naive http.Client opens up one persistent connection to the hostname, and then directs all requests to the same IP address internally. Granted, the client will open up concurrent connections and potentially bind connections to another host, but there's no predictability in this behavior.

Having a RoundTripper that was intelligent enough to do a DNS lookup, then round robin, or other load balancing algorithms across the IPs would be super nice.

I understand this isn't necessarily in the scope of connect-go since this is sorta being outsourced to the larger Go community by adopting net/http as the interface, but I think having some official RoundTripper provided by connect-go would be super useful.

I'm going to be working on a POC for this, unrelated to connect-go with not necessarily the intention for it to be upstreamed, but if we can come to agreements on if this is something connect-go should provide (even in a separate repo), I'd be happy to help contribute there.

In general, I personally want a RoundTripper for generic HTTP, which would happen to be also ideal within the context of connect-go. Right now, this is the barrier of entry to adopting connect-go to replace our gRPC clients. We've already replaced our gRPC servers. :)

I'm happy to provide more insight or suggestions as needed.

[akshayjshah]

🎉 We intended to do exactly this, right down to focusing on the http.RoundTripper interface! We haven't started work yet, so the timing is perfect - PlanetScale's requirements can help inform our design.

I think it probably makes sense to keep all this in a separate repository - it's useful for any HTTP client, and I bet we'll end up with some dependencies on connect-grpchealth-go.

Some questions:

• I have a soft spot for load balancing algorithms. I'd probably start with round robin & fewest pending but I'd love to get to more interesting approaches eventually. Does that work for you, or do you depend on something like peak EWMA or sophisticated subsetting today?
• I'd prefer to start with DNS, as you suggested, despite all the downsides of such a heavily cached, poll-based system - it's the lowest common denominator of discovery. Does that work for your use case, vs. some more k8s-specific approach? (I'm a relative Kubernetes novice, but there must be some high-level API that's effectively a watch on the underlying etcd key.)
• Discovery naturally leads to health checking, especially since DNS records are often stale. Would you want gRPC-specific health checks, classic HAProxy-style GETs, something else, or no health checks at all?

In short, yes we're planning on this, and yes we'd love some input from real Connect users.

[mattrobenolt]

We intended to do exactly this, right down to focusing on the http.RoundTripper interface! We haven't started work yet, so the timing is perfect - PlanetScale's requirements can help inform our design.

This is great, would love to help out here however I can. :) And glad we're on the same page on where to start.

I think it probably makes sense to keep all this in a separate repository - it's useful for any HTTP client, and I bet we'll end up with some dependencies on connect-grpchealth-go.

Agreed, in my case, I also would like this as a generic HTTP handler. I think overall this is a useful tool to standalone.

I have a soft spot for load balancing algorithms. I'd probably start with round robin & fewest pending but I'd love to get to more interesting approaches eventually. Does that work for you, or do you depend on something like peak EWMA or sophisticated subsetting today?

Yeah, tbh, we only need Round Robin, and it's what we're using today in gRPC. More is better, but I don't particularly have a need personally.

I'd prefer to start with DNS, as you suggested, despite all the downsides of such a heavily cached, poll-based system - it's the lowest common denominator of discovery. Does that work for your use case, vs. some more k8s-specific approach? (I'm a relative Kubernetes novice, but there must be some high-level API that's effectively a watch on the underlying etcd key.)

Agreed, we just use A records paired with a headless service in kube. So what that means is each pod gets a unique IP address which is surfaced in the DNS record. This can also be done with SRV records, so I'd say DNS support with A + SRV would cover most use cases. Having the discovery mechanism be pluggable would be ideal.

Discovery naturally leads to health checking, especially since DNS records are often stale. Would you want gRPC-specific health checks, classic HAProxy-style GETs, something else, or no health checks at all?

I hadn't thought about this, since this opens the door to a lot more complexity. In gRPC today, we don't use any active health checking. We rely on the DNS records to add/remove IPs as things become unhealthy, and kubernetes ends up being the source of truth. I think some primitive stuff would be nice to have, and just go to "next" IP based on actual network level errors. If the IP itself cannot be reached, mark dead, and try the next.

With all this said, I think the gRPC side of things are lacking too and we've done a number of hacks there as well. One of them being DNS refreshing. Once gRPC did a Dial() and resolved IPs, it never re-resolves until there are no healthy subconnections. This is a pretty... not very open repository, but here's an example of our substituted DNS resolver for gRPC: https://github.com/planetscale/psdb/blob/main/core/resolver/resolver.go to simply support refreshing DNS records periodically.

It's also common that gRPC pairs with a connection pooler of it's own as a pool of higher level gRPC objects, so there's mess like this. https://github.com/planetscale/psdb/blob/main/core/pool/pool.go

I have lots of opinions here. :)

[mattrobenolt]

And just for additional context, our other use case for a generic HTTP RoundTripper that supports this functionality is an HTTP proxy component. In our case, we run an effectively, reverse proxy frontend for gRPC that handles connections from clients, then routes to the correct backend internally based on Authorization headers, etc. This in turn, is effectively a transparent HTTP proxy, but in gRPC land this was a mess. Adding connect-go into the mix here makes this pretty difficult to port over currently, but instead making it a generic HTTP proxy and just have our ServeHTTP handle... any HTTP connection would be ideal. Similar to the httputil.NewSingleHostReverseProxy except, we'd shove this load balanced RoundTripper into the ReverseProxy instead and it'd work the same as using it in connect-go.

[flymedllva]

Will there be any alternative to channelz? I would like to see up-to-date information about the work of connections in the likeness of this https://github.com/rantav/go-grpc-channelz

[akshayjshah]

Will there be any alternative to channelz? I would like to see up-to-date information about the work of connections in the likeness of this https://github.com/rantav/go-grpc-channelz

No, there won't: connect-go delegates to the standard library's HTTP implementation, which manages TCP connections under the hood. net/http clients can get many similar insights from net/http/httptrace - there's no need for connect-go to get involved.

[haunt98]

Will it support scheme dns:/// to load balance?

[akshayjshah]

@cyriltovena is using connect-go in Grafana Labs' Phlare product, and also ran into a situation where a library like this would be useful.

[mattrobenolt]

I've been working on making my version a bit more... good, and capable of being consumed by the public, just has been taking time.

[akshayjshah]

@mattrobenolt and anyone else following this issue, @jhump and @jchadwick-buf are now actively working on this. We've got some internal use cases, and our Knit project is running into many of the same reverse proxy problems as PlanetScale.

Our intention is to:

• Do this work as a completely separate package that constructs net/http Clients and/or RoundTrippers. Clients should have defaults suited to intra-region, service-to-service use:
  • Disable cookie jars and redirects by default.
  • Support the HTTP_PROXY and HTTPS_PROXY environment variables by default.
  • In addition to the stdlib's redirect-handling function pointer, support a simpler "follow N redirects" option.
  • Encourage setting a timeout.
  • Eventually (but probably not in v1) make httptrace more discoverable.
• Make h2c easier and safer for clients. Rather than completely bypassing TLS, we can support a URL scheme like h2c://api.mycompany.com that uses h2c just for that pool of connections.
• Build a pluggable, layered set of discovery and load balancing abstractions with reasonable defaults.
  • Pluggable discovery, defaulting to DNS A records with periodic refreshes. We might add SRV record support after v1. The APIs we expose should enable users to plug in other discovery mechanisms like xDS, local hostfiles, or custom ZooKeeper setups.
  • Pluggable subsetting, defaulting to using all the discovered IPs. We'd like the subsetting APIs to enable simple use cases, like a fixed subset size (with some basic mechanisms to help each client choose a distinct subset). The APIs may be flexible enough to support complex approaches like Twitter's deterministic aperture or Uber's dynamic subsetting.
  • Pluggable active health checking, defaulting to none. The APIs should be flexible enough for users to opt into gRPC health checks, simple GET /health checks, or anything in between.
  • Pluggable load balancing. We'd like to implement at least random, round-robin, and fewest pending, with enough flexibility for users to implement more exotic balancers like peak EWMA. By default, clients should use one of the built-in algorithms.

We'll inevitably come across situations where the net/http APIs make handling some scenarios awkward or impossible. In those situations, we'll either back off and leave the problem unsolved (after all, people manage to use net/http in all sorts of environments today!) or we'll work with other interested parties to propose improvements to the standard library. At GopherCon last year, @rhysh expressed some interest in this - Twitch has apparently run into many sharp edges while using Twirp, especially with HTTP/2.

[jhump]

Another thing we talked about, though maybe it would be in a post-v1 version, is configuring a minimum number of connections. This is useful for cases where address resolution returns few addresses -- or more to the point, exactly one, like when using virtual IPs and hardware load balancers.

For HTTP/layer-7 load balancers, it's not a real concern. But when the VIP is effectively a TCP load balancer (which happens to be the case for typical Kubernetes Service IPs), it's a real issue if resolving to one IP also means creating only one connection. When HTTP/2 is used in this case -- so all requests get sent as concurrent streams on the same connection -- you have effectively no load distribution; everything is sent to a single backend. Having a minimum number of connections (configurable by backend host name or address, for example) could work around this: if you create three connections to the same VIP, chances are reasonable that they are connected to different backends. (That's the whole point of the TCP load balancer after all.)

I've done work on this sort of load balancing in the past, and it is rife with edge cases. In particular, it is still possible for all connections, regardless of how many, to end up on the same backend host. (This could happen, for example, if health checks in the load balancer were not perfectly configured and/or a rolling deploy/restart proceeded too quickly.) The "easiest", or dumbest, way to combat that is to do random reconnections: every so often pick a random connection from the pool, close it, and open a new one to replace it. Even better is if they can be non-random, like having servers add a response header to all responses with some notion of their "identity". Then the client can actually tell what kind of backend diversity it has in its connections, and use that info to make better choices about how often to reconnect and which connections to close (e.g. ones whose corresponding backend are over-represented).

[jhump]

Final related tidbit: it would be ideal if we could override the "maximum concurrent streams" in the server's settings frame -- like to tell the client to only allow a single concurrent stream per connection. With that, it would behave effectively like HTTP 1.1 connection pools, which don't have quite as many load balancing pathologies due to the fact that they naturally create multiple connections (and thus tend to have better backend diversity for better load distribution).

But this would likely be something we'd try to get fixed/added in golang.org/x/net/http2 rather than something we'd provide in a separate library.

[mattrobenolt]

👋 I am going to go through everything here tomorrow and give all the feedback I can based on my experiences so far. This is exciting.

[akshayjshah]

We've just cut the first public release of httplb - it brings configurable load balancing, name resolution, and subsetting to net/http clients! We're looking forward to your feedback over in that repository :)