grpc-dotnet icon indicating copy to clipboard operation
grpc-dotnet copied to clipboard

How to configure a gRPC client to allow OpenShift (HAProxy) to balance requests?

Open SeminDM opened this issue 3 months ago • 0 comments

My gRPC service is deployed in the OpenShift. I configured a route in such way:

kind: Route
apiVersion: route.openshift.io/v1
spec:
  tls:
    termination: reencrypt
    certificate: <...>
    key: <...>
    caCertificate: <...>
    destinationCACertificate: <...>

Such TLS termination allows HAProxy to spread requests between pods. I had used "Grpc 2.46.6" for client and service LB worked pretty well. But recently I've changed my client using "Grpc.Net.Client 2.63.0" and LB fails - HAProxy sends requests to single pod.

Question: how to configure a dotnet gRPC client for successful service load balancing?

Service:

  builder.WebHost.ConfigureKestrel(options =>
  {
  	options.ListenAnyIP(settings.Http2Port, listenOptions =>
  	{
  		// Http2 and https for gRPC.
  		listenOptions.Protocols = HttpProtocols.Http2;
  		listenOptions.UseHttps(settings.PathToPfx, settings.PfxPassword);
  	});
  	options.ListenAnyIP(settings.Http1Port, listenOptions =>
  	{
  		// Http1 for Prometheus and OpenShift probes.
  		listenOptions.Protocols = HttpProtocols.Http1;
  	});
  	options.Limits.MaxRequestBodySize = null;
  });
  builder.Services.AddGrpc(options =>
  {
  	options.EnableDetailedErrors = true;
  	options.MaxReceiveMessageSize = int.MaxValue;
  	options.MaxSendMessageSize = int.MaxValue;
  });
  var app = builder.Build();
  app.MapGrpcService<GrpcService>();
  app.MapGrpcService<GrpcStreamService>();

Grpc client:

 var options = new()
  {
     new(ChannelOptions.MaxSendMessageLength, int.MaxValue),
     new(ChannelOptions.MaxReceiveMessageLength, int.MaxValue),
     new("grpc.max_metadata_size", 10 * 1024 * 1024),
     new("grpc.enable_http_proxy", 0),
     new("grpc.lb_policy_name", "round_robin")
  };
  ChannelCredentials credentials = new SslCredentials(Resource.RootCert);
  var asyncAuthInterceptor = new AsyncAuthInterceptor((_, metadata) =>
  {
  	var spn = $"HTTP/{address}";
  	var token = KerberosTokenProvider.GetToken(spn);
  	metadata.Add("authorization-bin", token);
  	return Task.CompletedTask;
  });
  credentials = ChannelCredentials.Create(credentials, CallCredentials.FromInterceptor(asyncAuthInterceptor));
  _channel = new Channel(address, credentials, options);
  Client = CreateClient(_channel);

Grpc.Net.Client client.

  var credentials = CallCredentials.FromInterceptor((_, metadata) =>
  {
  	var spn = $"HTTP/{address}";
  	var token = KerberosTokenProvider.GetToken(spn);
  	metadata.Add("authorization-bin", token);
  	return Task.CompletedTask;
  });  
  var options = new GrpcChannelOptions
  {
  	MaxReceiveMessageSize = int.MaxValue,
  	MaxSendMessageSize = int.MaxValue,
  	ThrowOperationCanceledOnCancellation = true,
  	Credentials = ChannelCredentials.Create(ChannelCredentials.SecureSsl, credentials),
  	HttpHandler = new SocketsHttpHandler
  	{
  		EnableMultipleHttp2Connections = true,
  		UseProxy = false,
  	}
  };
  _channel = GrpcChannel.ForAddress($"https://{address}", options);
  Client = CreateClient(_channel);

SeminDM avatar May 27 '24 16:05 SeminDM