How to access Kubernetes GRPC Ingress calls with prefix?

[GautamSinghania]

We are using GRPC Ingress with Kubernetes and multiple paths. Here is the yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-grpc
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: GRPC
    nginx.ingress.kubernetes.io/proxy-body-size: 64m
    nginx.ingress.kubernetes.io/ssl-redirect: 'true'
    nginx.ingress.kubernetes.io/ssl-passthrough: 'true'
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
  - host: xxx.xxx.internal
    http:
      paths:
      - path: /(.*)
        pathType: Prefix
        backend:
          service:
            name: service-1
            port:
              number: 5001
      - path: /prefix-2/(.*)
        pathType: Prefix
        backend:
          service:
            name: service-2
            port:
              number: 5001
  tls:
  - secretName: grpc-tls
    hosts:
    - xxx.xxx.internal

We have the proto file for this, and tried hitting the services using

grpcurl -d '<<DATA>>' -proto <<PROTO_FILE>> -insecure xxx.xxx.internal:443 tensorflow.serving.PredictionService.Predict

We are able to hit service-1, which does not have any prefix. However, we are not able to hit service-2. We tried using xxx.xxx.internal:443/test and /test/tensorflow.serving.PredictionService.Predict, but none of them work.

We know that there is a Go script that can access this. Can you do a similar thing for grpcurl?

[jhump]

@GautamSinghania, does this mean the URLs for service-2 require a path prefix of /prefix-2/? That is generally not valid in gRPC.

The protocol spec for gRPC explicitly states that the path should be like so:

"/" Service-Name "/" {method name}

For this reason, none of the official gRPC clients will be able to access the service you are configuring since base paths are not supported. What kind of client are you using where you can make use of endpoints like that?

[GautamSinghania]

We were using Go. I have asked my colleague working on this to create a script for testing purposes, will share that here once complete.

[GautamSinghania]

As a standards question, what would be your suggestion to using multiple services under the same Ingress endpoint?

[jhump]

what would be your suggestion to using multiple services under the same Ingress endpoint?

Two ways

1. The simplest is to dispatch to one or the other based on the fully-qualified service name. So one service's ingress could enumerate all of the RPC services it exposes. A different service has an ingress that lists different service names. Obviously, having two services implement the same RPC service name will not work. (Though this can be problematic for other reasons, and is usually easily avoided.)
2. Use vhosts/sub-domains. This allows for multiple services exposing the same service name. This is how Google Cloud works, where multiple sub-domains of googleapis.com (firebase, datastore, bigtable, appengine, etc) all implement the long-running operations service.

[thunder7553]

@GautamSinghania, does this mean the URLs for service-2 require a path prefix of /prefix-2/? That is generally not valid in gRPC.

The protocol spec for gRPC explicitly states that the path should be like so:

"/" Service-Name "/" {method name}

For this reason, none of the official gRPC clients will be able to access the service you are configuring since base paths are not supported. What kind of client are you using where you can make use of endpoints like that?

I face the same problem - I do have a .NET 6 application (server and client), and the customer demands to have it in https://somehostname/<app>/ - would be the prefix. From what I understand here, it would be the "Request matching based on Prefix" A28

I did not follow gRPC development until now, so I do not know how this RFCs are applied. But from what I understand, it should be supported in the official clients.

It is my understanding, that those gRPC (A27 and A28) are talking about the SERVER SIDE - something about envoy proxy, and generic configuration. The crucial point, to me, is that this is a common use case, and those things happen in real life.

[thunder7553]

FWIW, how to access a GRPC service hosted in a directory in .NET 6 is documented on learn.microsoft.com

[jhump]

would be the prefix. From what I understand here, it would be the "Request matching based on Prefix" A28

@thunder7553, that proposal is for configuring XDS clients to support path matching for routing. But that does not mention anything about custom path prefixes. The gRPC Go client still has no option for specifying a base URI path, and the HTTP/2 wire spec still has no mention of support for a custom path prefix. FWIW, they are not server-side: XDS is a protocol for client-side load balancing, so it configures how clients should route requests to different discovered backends.

FWIW, how to access a GRPC service hosted in a directory in .NET 6 is documented on learn.microsoft.com

Note the big warning at the top of that page.

[thunder7553]

Thanks for the info, you are right indeed. I have to dig into that.

[renevall]

@thunder7553 This is what worked for me:

apiVersion: v1
kind: Service
metadata:
  name: "places"
  labels:
    app: "places"
    monitor: "true"
spec:
  type: NodePort
  ports:
    - name: tcp
      protocol: TCP
      port: 80
      targetPort: 50051
  selector:
    app: "places"

--
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
  name: "tracker"
spec:
  ingressClassName: nginx
  rules:
  - host: "dev.something.com"
    http:
      paths:
      - path: /org.places.v1.PlacesService # use package.ServiceName
        pathType: Prefix
        backend:
          service:
            name: "dev"
            port:
              number: 80
  tls:
    - hosts:
        - "dev.something.com"
      secretName: "dev.something.com-tls"

Then this will work:

grpcurl -import-path path/to/proto -proto service.proto -v -d '{"id":"018bd0c9-1ef2-788b-945f-a437e28b2a1c"}' \
    dev.something.com:443 org.places.v1.PlacesService/GetPlace

I have not managed to make the reflection work. This is because nginx gets /grpc.reflection.v1.ServerReflection/ServerReflectionInfo as the path. If we were able to get /org.places.v1.PlacesService/grpc.reflection.v1.ServerReflection/ServerReflectionInfo it would work since then nginx would know what path to take.

This is what you would see in the ingress logs:

111.77.196.166 - - [07/Jan/2024:05:37:51 +0000] "POST /grpc.reflection.v1.ServerReflection/ServerReflectionInfo HTTP/2.0" 499 0 "-" "grpcurl/1.8.9 grpc-go/1.57.0" 165 10.715 [upstream-default-backend] [] - - - - 2fe2729a167c69404b3707d59f658e2a <---- if you attempt reflection, it fails
111.77.196.166 - - [07/Jan/2024:05:39:22 +0000] "POST /org.places.v1.PlacesService/GetPlace HTTP/2.0" 200 0 "-" "grpcurl/1.8.9 grpc-go/1.57.0" 159 0.005 [development-places-80] [] 10.64.0.22:50051 0 0.006 200 f3a2aaa701016d21b246498657e1b87e
111.77.196.166 - - [07/Jan/2024:05:40:06 +0000] "POST /org.places.v1.PlacesService/GetPlace HTTP/2.0" 200 0 "-" "grpcurl/1.8.9 grpc-go/1.57.0" 159 0.010 [development-places-80] [] 10.64.0.22:50051 0 0.010 200 71eef1e2546b09db6139bd6c7a67102f

CC @jhump

[VenkateshSrini]

@thunder7553 This is what worked for me:

apiVersion: v1
kind: Service
metadata:
  name: "places"
  labels:
    app: "places"
    monitor: "true"
spec:
  type: NodePort
  ports:
    - name: tcp
      protocol: TCP
      port: 80
      targetPort: 50051
  selector:
    app: "places"

--
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
  name: "tracker"
spec:
  ingressClassName: nginx
  rules:
  - host: "dev.something.com"
    http:
      paths:
      - path: /org.places.v1.PlacesService # use package.ServiceName
        pathType: Prefix
        backend:
          service:
            name: "dev"
            port:
              number: 80
  tls:
    - hosts:
        - "dev.something.com"
      secretName: "dev.something.com-tls"

Then this will work:

grpcurl -import-path path/to/proto -proto service.proto -v -d '{"id":"018bd0c9-1ef2-788b-945f-a437e28b2a1c"}' \
    dev.something.com:443 org.places.v1.PlacesService/GetPlace

I have not managed to make the reflection work. This is because nginx gets /grpc.reflection.v1.ServerReflection/ServerReflectionInfo as the path. If we were able to get /org.places.v1.PlacesService/grpc.reflection.v1.ServerReflection/ServerReflectionInfo it would work since then nginx would know what path to take.

This is what you would see in the ingress logs:

111.77.196.166 - - [07/Jan/2024:05:37:51 +0000] "POST /grpc.reflection.v1.ServerReflection/ServerReflectionInfo HTTP/2.0" 499 0 "-" "grpcurl/1.8.9 grpc-go/1.57.0" 165 10.715 [upstream-default-backend] [] - - - - 2fe2729a167c69404b3707d59f658e2a <---- if you attempt reflection, it fails
111.77.196.166 - - [07/Jan/2024:05:39:22 +0000] "POST /org.places.v1.PlacesService/GetPlace HTTP/2.0" 200 0 "-" "grpcurl/1.8.9 grpc-go/1.57.0" 159 0.005 [development-places-80] [] 10.64.0.22:50051 0 0.006 200 f3a2aaa701016d21b246498657e1b87e
111.77.196.166 - - [07/Jan/2024:05:40:06 +0000] "POST /org.places.v1.PlacesService/GetPlace HTTP/2.0" 200 0 "-" "grpcurl/1.8.9 grpc-go/1.57.0" 159 0.010 [development-places-80] [] 10.64.0.22:50051 0 0.010 200 71eef1e2546b09db6139bd6c7a67102f

CC @jhump

I my both client and server are in .NET. I have the same issue when exposing the server through headless service. Does the solution that you offered works for headless service also?

[jhump]

This is because nginx gets /grpc.reflection.v1.ServerReflection/ServerReflectionInfo as the path. If we were able to get /org.places.v1.PlacesService/grpc.reflection.v1.ServerReflection/ServerReflectionInfo it would work since then nginx would know what path to take.

But then the server would have no idea how to handle it. I'm afraid the gRPC protocol does not describe any support for custom path prefixes. The recommendation is usually to route by service name (as you are doing) or to use sub-domains if you need to route the same service name to multiple backends.

[atifhafeez]

@jhump

Is it documented somewhere that gRPC protocol does not support custom path prefixes?

I am also facing the problem where if i define a custom path (instead of /, i use /atif/app1/), it does not work. Only if i just use path: /, the requests reaches server and i get a response.

Regards!

[jhump]

@atifhafeez, the gRPC protocol spec makes it pretty clear that the path must /service-name/method-name: https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md#requests

Currently, grpcurl is built on top of the official gRPC Go implementation, which does not provide the ability to send requests with custom URL path prefixes. In fact, if you look at all of the official (Google-provided) gRPC client libraries, none of them allow that. So making such a change to grpcurl would, unfortunately, require replacing or re-writing the transport implementation, which is a non-trivial lift.

[VenkateshSrini]

@thunder7553 This is what worked for me:

apiVersion: v1
kind: Service
metadata:
  name: "places"
  labels:
    app: "places"
    monitor: "true"
spec:
  type: NodePort
  ports:
    - name: tcp
      protocol: TCP
      port: 80
      targetPort: 50051
  selector:
    app: "places"

--
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
  name: "tracker"
spec:
  ingressClassName: nginx
  rules:
  - host: "dev.something.com"
    http:
      paths:
      - path: /org.places.v1.PlacesService # use package.ServiceName
        pathType: Prefix
        backend:
          service:
            name: "dev"
            port:
              number: 80
  tls:
    - hosts:
        - "dev.something.com"
      secretName: "dev.something.com-tls"

Then this will work:

grpcurl -import-path path/to/proto -proto service.proto -v -d '{"id":"018bd0c9-1ef2-788b-945f-a437e28b2a1c"}' \
    dev.something.com:443 org.places.v1.PlacesService/GetPlace

I have not managed to make the reflection work. This is because nginx gets /grpc.reflection.v1.ServerReflection/ServerReflectionInfo as the path. If we were able to get /org.places.v1.PlacesService/grpc.reflection.v1.ServerReflection/ServerReflectionInfo it would work since then nginx would know what path to take. This is what you would see in the ingress logs:

111.77.196.166 - - [07/Jan/2024:05:37:51 +0000] "POST /grpc.reflection.v1.ServerReflection/ServerReflectionInfo HTTP/2.0" 499 0 "-" "grpcurl/1.8.9 grpc-go/1.57.0" 165 10.715 [upstream-default-backend] [] - - - - 2fe2729a167c69404b3707d59f658e2a <---- if you attempt reflection, it fails
111.77.196.166 - - [07/Jan/2024:05:39:22 +0000] "POST /org.places.v1.PlacesService/GetPlace HTTP/2.0" 200 0 "-" "grpcurl/1.8.9 grpc-go/1.57.0" 159 0.005 [development-places-80] [] 10.64.0.22:50051 0 0.006 200 f3a2aaa701016d21b246498657e1b87e
111.77.196.166 - - [07/Jan/2024:05:40:06 +0000] "POST /org.places.v1.PlacesService/GetPlace HTTP/2.0" 200 0 "-" "grpcurl/1.8.9 grpc-go/1.57.0" 159 0.010 [development-places-80] [] 10.64.0.22:50051 0 0.010 200 71eef1e2546b09db6139bd6c7a67102f

CC @jhump

I my both client and server are in .NET. I have the same issue when exposing the server through headless service. Does the solution that you offered works for headless service also?

Can you please suggest how I go about this for headless service