Not able to route grpc stream to "grpc://" destination

[tommyalatalo]

I'm trying to use the new gRPC proxy from fabio 1.5.11 but I'm having a bit of an issue. My program is written in go and uses the native grpc client, and the addresses I've been giving it have always been in the format ip:port. But now when I'm trying to use fabio grpc proxy I see that the address routed to the destination grpc://ip:port. I've never seen this "grpc://" prefix before and my client fails to connect and stream to that address. Why is the "grpc://" there, and how can I work around it?

[magiconair]

@andyroyle can you maybe help here?

[andyroyle]

The grpc:// prefix is purely an implementation detail for the Fabio config language. Since the grpc proxy is separate from the http proxy (in spite of the fact that grpc runs over http2), Fabio needs to be able to differentiate between http routes and grpc routes, hence the proto=grpc option.

A URL in the routing table that looks like grpc://127.0.0.1:3456 is normal. When connecting to the downstream service Fabio will just use the host and port.

The clients and backend services should be completely unaware of the internal grpc:// prefix.

Can you post more information about your setup and the errors you are seeing?

[tommyalatalo]

The grpc:// prefix is purely an implementation detail for the Fabio config language. Since the grpc proxy is separate from the http proxy (in spite of the fact that grpc runs over http2), Fabio needs to be able to differentiate between http routes and grpc routes, hence the proto=grpc option.

A URL in the routing table that looks like grpc://127.0.0.1:3456 is normal. When connecting to the downstream service Fabio will just use the host and port.

The clients and backend services should be completely unaware of the internal grpc:// prefix.

Can you post more information about your setup and the errors you are seeing?

I'm running a grpc server in a docker container registered as a service in a consul cluster which fabio is part of too. I'm dialing using c, err := grpc.Dial(serverAddr, grpc.WithInsecure(), grpc.WithBackoffMaxDelay(5*time.Second)) The server is running a bidirectional stream, but the client stops dead since it cant connect (no error output, just a println describing which step the client streaming is at before it fails connecting), this happens when I point the client to fabio.service.consul:8888/grpcserver/ (serverAddr above).

I can match this behavior exactly by pointing my client manually to ip:port (it works) and then to grpc://ip:port, and it fails in the same manner as above when targeting the consul dns fabio address.

The consul service is basically set up like this in my Nomad job for the container: service { name = "grpcserver" port = "grpc" tags = ["grpc", "urlprefix-/grpcserver/ proto=grpc "] check { name = "grpcserver tcp" type = "tcp" interval = "10s" timeout = "2s" } }

I haven't specified the RPCs as the endpoints here, but I tried that too, and it makes no difference. From my end it seems that the implementation doesn't work the way you're saying it does, if fabio would have resolved the path to just ip:port then my client should definitely be able to connect, instead now it seems very much like it resolves to the "grpc://" path, or something else thats not connectable.

Here is my fabio job for completeness: `job "fabio" { datacenters = ["dc"] type = "system"

update { stagger = "60s" max_parallel = 1 }

group "fabio" {

task "fabio" {
  driver = "docker"

  config {
    image = "fabiolb/fabio:latest"
    dns_servers = ["169.254.1.1"]
    network_mode = "host"
  }

  env {
    registry_consul_addr = "169.254.1.1:8500"
    proxy_addr = ":9999;proto=http,:8888;proto=grpc"
  }

  service {
    name = "fabio-http"
    tags = ["http", "load-balancer", "proxy"]
    port = "http"

    check {
      type     = "tcp"
      interval = "10s"
      timeout  = "2s"
    }
  }

  service {
    name = "fabio-ui"
    tags = ["ui", "load-balancer"]
    port = "ui"
    check {
      type     = "http"
      path     = "/"
      interval = "30s"
      timeout  = "3s"
    }
  }

  resources {
    cpu    = 500
    memory = 64
    network {
      mbits = 20
      port "http" { static = 9999 }
      port "ui" { static = 9998 }
      port "grpc" { static = 8888 }
    }
  }

}

} } `

That is to say unless something in my configuration above is incorrect? Can I provide you with more information?

[andyroyle]

Ah, I need to confirm but I am pretty sure that the urlprefix (i.e. /grpc-server) isn't supported by the grpc proxy.

I don't know whether it is possible to add support for it.

[tommyalatalo]

Ah, I need to confirm but I am pretty sure that the urlprefix (i.e. /grpc-server) isn't supported by the grpc proxy.

I don't know whether it is possible to add support for it.

That seems to be the opposite of what is shown on the Fabio Quickstart page (https://fabiolb.net/quickstart/) Where my use case would be the second from the top (see the list below), having a service specific route not pointing to a specific rpc.

urlprefix-/my.service/Method proto=grpc # method specific route urlprefix-/my.service proto=grpc # service specific route urlprefix-/my.service proto=grpcs # TLS upstream urlprefix-/my.service proto=grpcs grpcservername=my.service # TLS upstream with servername override urlprefix-/my.service proto=grpcs tlsskipverify=true # TLS upstream and self-signed cert

Am I understanding you correctly in that this doesn't work, or are you referring to something else when you say that the url prefix "/grpcserver" isn't supported? Is "my.service" supposed to be something specific in the examples above?

[andyroyle]

In the case of the my.service, it's specifically the grpc service name (grpc calls are routed by service and method name). Unfortunately arbitrary prefixes aren't supported in the way that they are for http/s proxying

The docs are unclear so that's a failure on my part.

The grpc service is defined by the package name and service name defined in your proto, e.g.

package foo

service bar {
  rpc Baz (BazReq) returns (BazResp) {}
  rpc Flarg (FlargReq) returns (FlargResp) {}
}

// message definitions

So in the above example, the service is foo.bar and the method is Baz or Flarg.

You can route using only the service name:

• urlprefix-/foo.bar proto=grpc

Or using both the service and method

• urlprefix-/foo.bar/Baz proto=grpc

Or using just a catch-all

• urlprefix-/ proto=grpc

[andyroyle]

That's not to say that arbitrary prefixes and rewriting and stripping and the like couldn't be supported, it's just the current implementation doesn't

[tommyalatalo]

Sorry for a delayed reply. I've been trying this out a bit, but I'm still not getting my streams through to the grpc server. I added "urlprefix-/protopkg/protoservice proto=grpc" to my tags, and tried routing using service name and the catchall ''/", but still nothing.

Using the urlprefix I'm targeting fabio.service.consul:8888/protopkg/protoservice I've set 8888 as the grpc listener port in fabio's nomad job as seen in my previous post, and the port is allocated to fabio as static.

Not sure what to try next at this point?

[aaronhurt]

Late to the conversation but looking at the code examples provided above it doesn't look like you are using the correct separator for the package and service. You have /protopkg/protoservice/ and the example shows /foo.bar with an example package of foo and a service bar. I assume that should make your tag urlprefix-/protopkg.protoservice proto=grpc.

[tommyalatalo]

Late to the conversation but looking at the code examples provided above it doesn't look like you are using the correct separator for the package and service. You have /protopkg/protoservice/ and the example shows /foo.bar with an example package of foo and a service bar. I assume that should make your tag urlprefix-/protopkg.protoservice proto=grpc.

I tried it with '.' as well, typoed it into my reply unfortunately. Is the route case sensitive?

[andyroyle]

As far as I am aware, yes the routing is case sensitive

[tommyalatalo]

This is still not working, I've set up my urlprefix as urlprefix-/protopkg.protoService proto=grpc and trying to stream to it at fabio.service.consul:8888/protopkg.protoService, but the grpc client still can't connect. Can I do some step-by-step verification to see that the grpc listener port is working to begin with, and then look further into why the grpc client isn't connecting?

[pschultz]

In the Go client do not add a path when dialing. gRPC will construct the path automatically based on the method call.

Use

grpc.Dial("fabio.service.consul:8888")

not

grpc.Dial("fabio.service.consul:8888/some-path")

[tommyalatalo]

I'm using this line to dial: c, err := grpc.Dial(grpcServerAddr, grpc.WithInsecure(), grpc.WithBackoffMaxDelay(5*time.Second)) where grpcServerAddr is "fabio.service.consul:8888", but it's still not connecting.

There are no limitations in fabio as to what kind of communication is supported for grpc? I'm trying to do a bidirectional stream, is that a problem?

A followup question to this is; how do I run two versions of the same grpc server with the same protobuf services with fabio, so that I'm able to send traffic deliberately to either one, v1 or v2? Just point to fabio.service.consul:8888 doesn't give me any control over which server I will reach?

[tommyalatalo]

All right! Finally got it to connect, you were right, the dial address needs to be EXACTLY "fabio.service.consul:8888". Even a trailing slash will prevent it from connecting, i.e. "fabio.service.consul:8888/"

This documentation needs to be updated, it's impossible to derive this from what is stated in the docs.

I'm really in need of an answer to my other question though which was how I can explicitly route traffic to a certain version of a service. Say I have service A with version 1 and 2 running side-by-side in blue/green or canary deployment, how can I route one client to v2 and keep the rest on v1 (i.e. I don't want to route percentages, I want to be able to point a certain client at the newer version of the service)?

Does this require url prefixes support? Like: fabio.service.consul:8888/v1/ fabio.service.consul:8888/v2/ Or can it be done some other way?

[tommyalatalo]

Any input to my last question here?

[aaronhurt]

@andyroyle provided some stub example code ... should be able to expand that to a small PoC. If you do it would be great to post it back here.

[andyroyle]

Sorry @tommyalatalo I've been on vacation this past week.

Yeah, it seems like the docs need updating. I'll hopefully get to it this week.

In terms of your question about prefixes, whilst they aren't currently supported, we essentially need to hook up the path stripping. In theory it might already be supported (if you add the strip=/v1 option to the tag), I've just never tested it.

I need to test to see how this works. Hopefully I will get to it soon.

[tommyalatalo]

Sorry @tommyalatalo I've been on vacation this past week.

Yeah, it seems like the docs need updating. I'll hopefully get to it this week.

In terms of your question about prefixes, whilst they aren't currently supported, we essentially need to hook up the path stripping. In theory it might already be supported (if you add the strip=/v1 option to the tag), I've just never tested it.

I need to test to see how this works. Hopefully I will get to it soon.

I tried some path stripping with grpc now, and it doesn't seem to work. I'm guessing implementing path stripping for grpc should be very similar to the http stripping, so how is hoping this would be almost a cut-and-paste update for the grpc side of things?

Really appreciate your help and support!

[tommyalatalo]

@andyroyle How long do you think it would be until you could get some path stripping implemented? Days, weeks, months?

[tommyalatalo]

@andyroyle By the way, I was thinking, is path stripping going to be enough to allow routing to different endpoints in this case?

Won't it end up like this:

fabio.service.consul:8888/v1/ strip=/v1 -> fabio.service.consul:8888/pkg.service
fabio.service.consul:8888/v2/ strip=/v2 -> fabio.service.consul:8888/pkg.service

Both will end up at fabio.service.consul:8888 and grpc resoplver will add the protobuf service after the port, and both paths will end up at the same grpc server? Isn't it essentially required that the fabio grpc protocol supports path prefixes that actually route to different host:port combinations? The problem at the moment being that all grpc requests are served through "fabio.service.consul:8888"?

So that

fabio.service.consul:8888/v1/ -> 10.10.10.10:1111 (grpc server version 1)
fabio.service.consul:8888/v2/ -> 10.10.10.10:2222 (grpc server version 2) 

Maybe I'm misreading your last post, and you're actually saying that in order to do the above you need to support path stripping as well to complement the prefixes.

[andyroyle]

Ah bum. You're right. Since the path is constructed by the proto client the. We can't just add arbitrary bits to it.

I suspect the only way to achieve what you want is to create two separate proto services pkg.service.v1 and pkg.service.v2 and register them separately.

[aaronhurt]

You could route via hostname instead of path ... create multiple A records (or CNAME entries) that resolve to the address of your fabio server. Then you could urlprefix-svc1.domain.com and urlprefix-svc2.domain.com to different backends.

[andyroyle]

The problem there is that the host matching doesn't work for GRPC. GRPC uses http2, for which the host header isn't required, so at routing time, we don't have any information as to what host the request came in on. All we know is what server config can tell us i.e. the IP we bound to or the server name that is in the certificate we are using (good luck if you are using a wildcard cert)

[aaronhurt]

Hrm ... HTTP/2 does require SNI by specification though. So, if we're not already using SNI to match host routes on HTTP/2 we should. I'll look into the code in more detail when I get a chance.

[andyroyle]

Presumably SNI is only required for TLS, so that wouldn't cover non-TLS deployments. Still, it's better than nothing I guess.

[aaronhurt]

Ohh right, that would only affect HTTP/2 and/or GRPC over TLS but yes ... it would be better than nothing in the absence of a host header. Ideally we would have a documented and logical search order. Something like HOST header, SNI, CN from cert, IP, ???

[tommyalatalo]

@andyroyle @leprechau Having two separate proto services is not viable, can't have all devs creating duplicates of services for upgrade purposes, that would be incredibly error prone.

Bear with me if I'm not quite on the same wavelength as you guys, but what I gather is that this could be solved using TLS certified connections, how would that work? I'm looking at setting up TLS for our traffic in the future anyway.

Also curious about the host header, is it possible to use that with GRPC and add it to the requests for routing purposes? I have full control over the clients so if that would be a way forward I would be interested in that too.

A thought that I had earlier, which might not be feasible at all, was that my particular issue could perhaps solvable by having the option to set up multiple listener ports for the same protocol, which would have unique routes. Say port 8888 could route to v1 of a service and port 7777 could route to v2. This is a bit ugly and the use cases for it in general probably pretty slim, so maybe leave it as just a thought.

[tommyalatalo]

I'm trying to set up TLS with SNI, but the docs are lacking a bit here, can you give an example of how to use this line: urlprefix-/ proto=grpcs grpcservername=my.service.hostname is this also supposed to be protopkg.protoService.hostname and what should the hostname be in this case? A consul dns address, or something else?

[andyroyle]

the grpcservername option was added for use with TLS backends. Fabio connects to backends using <ip>:<port>, so if the certificate presented by your backend doesn't include an IP SAN, then you can set grpcservername to specify the serverNameOverride field in the tls config: https://github.com/fabiolb/fabio/blob/master/proxy/grpc_handler.go#L245

An example would be:

urlprefix-/ proto=grpcs grpcservername=grpcapi.service.consul