How to configure service mesh to use gRPC protocol between services

[Shige99011]

Hi, I would like to configure consul connect to use gRPC protocol between services within the service mesh. But the sending request from a service to another seems to fail. How should it be configured?

Here is what I'm doing:

There are 2 services and they want to communicate by using gRPC. Service grpcserver: This is listening at the port 65001 with http2 protocol. Service client: This opens the channel for gRPC with http://grpcserver:65001. Then communicates with grpcserver. These are working on Docker compose environment at least but not working on Kubernetes with consul.

Here is my configuration for consul:

global:
  name: consul
 
client:
  enabled: true
  grpc: true
  exposeGossipPorts: true
 
ui:
  enabled: true
  service:
    enabled: true
    type: 'NodePort'
 
connectInject:
  enabled: true
  transparentProxy:
    defaultEnabled: true

connect:
  enabled: true

controller:
  enabled: true
   
syncCatalog:
  enabled: true
  toConsul: true
  toK8S: true
 
server:
  enabled: true
  replicas: 1
  bootstrapExpect: 1
  disruptionBudget:
    enabled: true
    maxUnavailable: 0

And the services are deployed with:

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: client
spec:
  protocol: grpc
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: grpcserver
spec:
  protocol: grpc

For each service deployment:

apiVersion: v1
kind: Service
metadata:
  name: grpcserver
spec:
  selector:
    app: grpcserver
  ports:
  - port: 80
    targetPort: 65001
    name: grpc
    protocol: TCP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grpcserver
  name: grpcserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grpcserver
  template:
    metadata:
      annotations:
        'consul.hashicorp.com/connect-inject': 'true'

      labels:
        app: grpcserver
    spec:
      containers:
      - name: grpcserver
        image: grpcserver:latest
        imagePullPolicy: Never 
        ports:
        - containerPort: 65001
          name: grpc
        env:
           - name: 'SERVICE_PORT'
             value: "65001"
---
apiVersion: v1
kind: Service
metadata:
  name: client
spec:
  selector:
    app: client
  ports:
  - port: 80
    targetPort: 65005
    name: grpc
    protocol: TCP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: client
  name: client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      annotations:
        'consul.hashicorp.com/connect-inject': 'true'
        'consul.hashicorp.com/connect-service-upstreams': 'grpcserver:65001'
      labels:
        app: client
    spec:
      containers:
      - name: client
        image: client:latest
        imagePullPolicy: Never #Need to pull from local registry?
        ports:
        - containerPort: 65005
        env:
           - name: 'SERVICE_PORT'
             value: "65001"
           - name: 'SERVER_ADDRESS'
             value: 'grpcserver'

This is the log form client service:

{"EventId":0,"LogLevel":"Information","Category":"Microsoft.Hosting.Lifetime","Message":"Application started. Press Ctrl\u002BC to shut down.","State":{"Message":"Application started. Press Ctrl\u002BC to shut down.","{OriginalFormat}":"Application started. Press Ctrl\u002BC to shut down."}}
{"EventId":0,"LogLevel":"Information","Category":"Microsoft.Hosting.Lifetime","Message":"Hosting environment: Production","State":{"Message":"Hosting environment: Production","envName":"Production","{OriginalFormat}":"Hosting environment: {envName}"}}
{"EventId":0,"LogLevel":"Information","Category":"Microsoft.Hosting.Lifetime","Message":"Content root path: /app","State":{"Message":"Content root path: /app","contentRoot":"/app","{OriginalFormat}":"Content root path: {contentRoot}"}}
{"EventId":9,"LogLevel":"Error","Category":"Microsoft.Extensions.Hosting.Internal.Host","Message":"BackgroundService failed","Exception":"Grpc.Core.RpcException: Status(StatusCode=\u0022Internal\u0022, Detail=\u0022Error starting gRPC call. HttpRequestException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.\u0022, DebugException=\u0022System.Net.Http.HttpRequestException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.    at System.Net.Http.HttpConnectionPool.ReturnHttp2Connection(Http2Connection connection, Boolean isNewConnection)    at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(HttpRequestMessage request)    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.ExecutionContextCallback(Object s)    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.MoveNext(Thread threadPoolThread)    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.MoveNext()    at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)    at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)    at System.Threading.Tasks.Task.FinishContinuations()    at System.Threading.Tasks.Task\u00601.TrySetResult(TResult result)    at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder\u00601.SetResult(TResult result)    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.ExecutionContextCallback(Object s)    at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.MoveNext(Thread threadPoolThread)    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.ExecuteFromThreadPool(Thread threadPoolThread)    at System.Threading.ThreadPoolWorkQueue.Dispatch()    at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()    at System.Threading.Thread.StartCallback() --- End of stack trace from previous location ---    at System.Threading.Tasks.TaskCompletionSourceWithCancellation\u00601.WaitWithCancellationAsync(CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.HttpClient.\u003CSendAsync\u003Eg__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)    at Grpc.Net.Client.Internal.GrpcCall\u00602.RunCall(HttpRequestMessage request, Nullable\u00601 timeout)\u0022)    at GrpcClient.ClientService.ExecuteAsync(CancellationToken stoppingToken) in /src/ClientService.cs:line 46    at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)","State":{"Message":"BackgroundService failed","{OriginalFormat}":"BackgroundService failed"}}
{"EventId":10,"LogLevel":"Critical","Category":"Microsoft.Extensions.Hosting.Internal.Host","Message":"The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.","Exception":"Grpc.Core.RpcException: Status(StatusCode=\u0022Internal\u0022, Detail=\u0022Error starting gRPC call. HttpRequestException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.\u0022, DebugException=\u0022System.Net.Http.HttpRequestException: An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.    at System.Net.Http.HttpConnectionPool.ReturnHttp2Connection(Http2Connection connection, Boolean isNewConnection)    at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(HttpRequestMessage request)    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.ExecutionContextCallback(Object s)    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.MoveNext(Thread threadPoolThread)    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.MoveNext()    at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)    at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)    at System.Threading.Tasks.Task.FinishContinuations()    at System.Threading.Tasks.Task\u00601.TrySetResult(TResult result)    at System.Runtime.CompilerServices.AsyncValueTaskMethodBuilder\u00601.SetResult(TResult result)    at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.ExecutionContextCallback(Object s)    at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.MoveNext(Thread threadPoolThread)    at System.Runtime.CompilerServices.AsyncTaskMethodBuilder\u00601.AsyncStateMachineBox\u00601.ExecuteFromThreadPool(Thread threadPoolThread)    at System.Threading.ThreadPoolWorkQueue.Dispatch()    at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()    at System.Threading.Thread.StartCallback() --- End of stack trace from previous location ---    at System.Threading.Tasks.TaskCompletionSourceWithCancellation\u00601.WaitWithCancellationAsync(CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)    at System.Net.Http.HttpClient.\u003CSendAsync\u003Eg__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)    at Grpc.Net.Client.Internal.GrpcCall\u00602.RunCall(HttpRequestMessage request, Nullable\u00601 timeout)\u0022)    at GrpcClient.ClientService.ExecuteAsync(CancellationToken stoppingToken) in /src/ClientService.cs:line 46    at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)","State":{"Message":"The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.","{OriginalFormat}":"The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted."}}
{"EventId":0,"LogLevel":"Information","Category":"Microsoft.Hosting.Lifetime","Message":"Application is shutting down...","State":{"Message":"Application is shutting down...","{OriginalFormat}":"Application is shutting down..."}}

What is wrong or missed?

My consul environment is helm: 0.41.1 consul: 1.11.3 Kubernetes: k3s version v1.21.3+k3s1 (1d1f220f)

Thanks for your help.

[t-eckert]

Thank you for this question, @Shige99011. Your configuration for Consul seems to be right. I'm not certain how your application itself behaves, but is there a potential issue with this mismatch of targeting the containerPort: 65005 while the SERVICE_PORT is 65001? I'm just seeing that in your client service and deployment.

[ishustava]

Hey @Shige99011

If you're using explicit upstreams, i.e. providing the connect-service-upstreams annotation, then you'll need to address your upstream service via localhost (set SERVER_ADDRESS to localhost).

Since you're enabling tproxy, I'd recommend omitting this annotation and seeing if that fixes your issue. If you omit it, you can use kube DNS names directly.

[Shige99011]

Thanks guys for your reply. I could confirm the communication by grpc has worked between the services anyway. yes, upstream annotations are not needed as transparent proxy is enabled but it seems to work even if there is that.

By the way, is it possible for envoy proxy to configure multiple ports? I would like to configure a service to have 2 kinds of port. (e.g. one is for grpc, another one is for http). The service has rest api for outside of service mesh and has grpc communication with the other services within the service mesh. According to the doc, it seems to be able to configure only one protocol for a service as ServiceDefaults, though..

[ishustava]

Hey @Shige99011

For multi-port support, we currently have a workaround documented here: https://www.consul.io/docs/k8s/connect#kubernetes-pods-with-multiple-ports

[Shige99011]

Thanks for the info. But it doesn't work yet although I tried it. The error says: For http(rest) connection: System.Net.Http.HttpRequestException: Connection refused (127.0.0.1:1234)

For grpc connection: Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: Connection refused (127.0.0.1:2234) SocketException: Connection refused", DebugException="System.Net.Http.HttpRequestException: Connection refused (127.0.0.1:2234)

My trial is as below. Is there any wrong or missing? Thanks for your help.

ServiceDefaults configuration:

# apiVersion: consul.hashicorp.com/v1alpha1
# kind: ServiceDefaults
# metadata:
#   name: client
# spec:
#   protocol: tcp
# ---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: grpcserver
spec:
  protocol: grpc

---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: grpcserver2
spec:
  protocol: http

Deployment configuration:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: grpcserver
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: grpcserver2
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: client
---

apiVersion: v1
kind: Service
metadata:
  name: grpcserver
spec:
  selector:
    app: grpcserver
  ports:
  - port: 80
    targetPort: 65001
    name: grpc
    protocol: TCP
---

apiVersion: v1
kind: Service
metadata:
  name: grpcserver2
spec:
  selector:
    app: grpcserver
  ports:
  - port: 80
    targetPort: 65000
    name: http
    protocol: TCP
---

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grpcserver
  name: grpcserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grpcserver
  template:
    metadata:
      annotations:
        'consul.hashicorp.com/connect-inject': 'true'
        'consul.hashicorp.com/transparent-proxy': 'false'
        'consul.hashicorp.com/connect-service': 'grpcserver,grpcserver2'
        'consul.hashicorp.com/connect-service-port': '65001,65000'
      labels:
        app: grpcserver
    spec:
      containers:
      - name: grpcserver
        image: grpcserver:latest
        imagePullPolicy: Never #Need to pull from local registry?
        ports:
        - containerPort: 65001
          name: grpc
        env:
           - name: 'GRPC_PORT'
             value: "65001"
      - name: grpcserver2
        image: grpcserver:latest
        imagePullPolicy: Never #Need to pull from local registry?
        ports:
        - containerPort: 65000
          name: http
        env:
           - name: 'HTTP_PORT'
             value: "65000"
      serviceAccountName: grpcserver
---

apiVersion: v1
kind: Service
metadata:
  name: client
spec:
  selector:
    app: client
  ports:
  - port: 80
    targetPort: 65005
    name: grpc
    protocol: TCP
---

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: client
  name: client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: client
  template:
    metadata:
      annotations:
        'consul.hashicorp.com/connect-inject': 'true'
        'consul.hashicorp.com/transparent-proxy': 'false'
        'consul.hashicorp.com/connect-service-upstreams': "grpcserver:2234,grpcserver2:1234"
      labels:
        app: client
    spec:
      containers:
      - name: client
        image: client:latest
        imagePullPolicy: Never #Need to pull from local registry?
        ports:
        - containerPort: 65005
        env:
           - name: 'SERVER_ADDRESS'
             value: '127.0.0.1'
           - name: 'SERVICE_PORT'
             value: '2234'
           - name: 'SERVICE_PORT_REST'
             value: '1234'

The consul configuration:

# Choose an optional name for the datacenter
global:
  name: consul

client:
  enabled: true
  grpc: true
  exposeGossipPorts: true


# Enable the Consul Web UI via a NodePort
ui:
  enabled: true
  service:
    enabled: true
    type: 'NodePort'

# Enable Connect for secure communication between nodes
connectInject:
  enabled: true
  transparentProxy:
    defaultEnabled: false

connect:
  enabled: true


# Enable CRD Controller
controller:
  enabled: true

# Automatically registers services in kubernetes to consul
syncCatalog:
  enabled: true
  toConsul: true
  toK8S: true

# Use only one Consul server for local development
server:
  enabled: true
  replicas: 1
  bootstrapExpect: 1
  disruptionBudget:
    enabled: true
    maxUnavailable: 0