Outputting `service` for @grpc/grpc-js

[faustbrian]

trafficstars

Hey, is there currently any way of outputting service definitions as code? Basically --ts_proto_opt=outputServices=grpc-js from https://github.com/stephenh/ts-proto but for this package.

Currently I'm using a mix of protobuf-es and ts-proto where ts-proto is used for our gRPC server. Would it be possible with some extensions or transpiler if it isn't possible out of the box?

[timostamm]

There is no built-in feature to generate service definitions that are compatible with @grpc/grpc-js, but something quite similar. In protobuf-es, services are represented as a ServiceType - an object with a name and a set of methods, each with information about the streaming type, and the type of the input and output message.

For example, let's take the following service definition in protobuf:

syntax="proto3";
package demo;

service ElizaService {
  // Say is a unary request demo. This method should allow for a one sentence
  // response given a one sentence request.
  rpc Say(SayRequest) returns (SayResponse) {}
}

// SayRequest describes the sentence said to the ELIZA program.
message SayRequest {
  string sentence = 1;
}

// SayResponse describes the sentence responded by the ELIZA program.
message SayResponse {
  string sentence = 1;
}

The corresponding ServiceType would be:

const ElizaService: ServiceType = {
  typeName: "demo.ElizaService",
  methods: {
    say: {
      name: "Say",
      I: SayRequest,
      O: SayResponse,
      kind: MethodKind.Unary,
    },
  }
};

@bufbuild/protoc-gen-es does not generate anything for services, but @bufbuild/protoc-gen-connect-web does. It happens to generate pretty much the example above, and @bufbuild/connect-web uses TypeScript's mapped types to turn it into a client:

import {createPromiseClient} from "@bufbuild/connect-web";

const client = createPromiseClient(ElizaService, ...);
client.say({ sentence: "Hello" }); // this is type-safe

The fun part is that you can simply use @bufbuild/protoc-gen-connect-web to generate service types, and turn them to @grpc/grpc-js clients and servers. We're making sure that this works in unit tests here and here. So connecting a few dots, you end up with a @grpc/grpc-js client:

import * as grpc from "@grpc/grpc-js";

const client = createGrpcClient(ElizaService, { 
   address: "localhost",
   channelCredentials: grpc.ChannelCredentials.createInsecure()
});

client.say({ sentence: "Hello" }, (err: grpc.ServiceError | null, value?: SayResponse) => {
  //       
});

This is just a proof of concept, but it should work pretty well. If you want to give it a try, let us know if you encounter any problems!

[faustbrian]

Thanks! That did indeed work after I copied https://github.com/bufbuild/connect-web/blob/3888ec37d7050863acefbb3b4a9c5e30367de8f6/packages/connect-web-test/src/nodeonly/create-grpc-definition.ts into my project. Are these functions exposed anywhere else outside of that package? It has a fair few testing-only dependencies so copy-paste seems more reasonable.

[timostamm]

Great!

The functions are not public because they only serve as a proof of concept. I understand that it would be pretty useful for you to have them in a public package? Let's keep this issue open then.

[faustbrian]

Since it works without any issues so far (all our tests are passing as they did with ts-proto) I think it would be nice to have them in a public package and documented since using them with grpc-js is probably a fairly common use case.

[fubhy]

Are you planning on having something like https://github.com/timostamm/protobuf-ts/tree/master/packages/grpc-backend as part of the connect stack (so basically connect-node or sth.)?

If not, I'd be keen to work on that after the schema validation thing.

That's one of the missing puzzle pieces for us to migrate our existing services over. We liked the "convenience" / simplicity of the service method wrappers it provided (simple return for unary methods, etc.) over the default grpc-js ones.

[timostamm]

Yes we do, Sebastian. I hope to share our roadmap and plans along with the first public commits for connect-node.

[jellydn]

There is no built-in feature to generate service definitions that are compatible with @grpc/grpc-js, but something quite similar. In protobuf-es, services are represented as a ServiceType - an object with a name and a set of methods, each with information about the streaming type, and the type of the input and output message.

For example, let's take the following service definition in protobuf:

syntax="proto3";
package demo;

service ElizaService {
  // Say is a unary request demo. This method should allow for a one sentence
  // response given a one sentence request.
  rpc Say(SayRequest) returns (SayResponse) {}
}

// SayRequest describes the sentence said to the ELIZA program.
message SayRequest {
  string sentence = 1;
}

// SayResponse describes the sentence responded by the ELIZA program.
message SayResponse {
  string sentence = 1;
}

The corresponding ServiceType would be:

const ElizaService: ServiceType = {
  typeName: "demo.ElizaService",
  methods: {
    say: {
      name: "Say",
      I: SayRequest,
      O: SayResponse,
      kind: MethodKind.Unary,
    },
  }
};

@bufbuild/protoc-gen-es does not generate anything for services, but @bufbuild/protoc-gen-connect-web does. It happens to generate pretty much the example above, and @bufbuild/connect-web uses TypeScript's mapped types to turn it into a client:

import {createPromiseClient} from "@bufbuild/connect-web";

const client = createPromiseClient(ElizaService, ...);
client.say({ sentence: "Hello" }); // this is type-safe

The fun part is that you can simply use @bufbuild/protoc-gen-connect-web to generate service types, and turn them to @grpc/grpc-js clients and servers. We're making sure that this works in unit tests here and here. So connecting a few dots, you end up with a @grpc/grpc-js client:

import * as grpc from "@grpc/grpc-js";

const client = createGrpcClient(ElizaService, { 
   address: "localhost",
   channelCredentials: grpc.ChannelCredentials.createInsecure()
});

client.say({ sentence: "Hello" }, (err: grpc.ServiceError | null, value?: SayResponse) => {
  //       
});

This is just a proof of concept, but it should work pretty well. If you want to give it a try, let us know if you encounter any problems!

Just FYI - This approach is working. I've implemented a simple on my monorepo just in case if someone need this.

[btiernay]

👋 Wanted to checkin on the status of this request. I see that @jellydn implemented this via https://github.com/jellydn/grpc-demo-monorepo/tree/main/apps/modern-hello/connect-node-extra and was wondering if something like that would be added to the project. Thanks!

[timostamm]

Hey Bob, this repository provides a foundation for protobuf in JavaScript and TypeScript, but it isn't concerned with networking at all. But it was designed to be easily built on and add networking functionality/code generators.

Our own take for this is Connect. We have already published connect-go and connect-web, and we are currently working on connect-node.

We realize that there's a use case for @grpc/grpc-js with @bufbuild/protobuf, but right now, our attention is focused on a first beta release of connect-node, which means we cannot publish and support integration with @grpc/grpc-js at the moment.

But the code we wrote as a proof of concept is technically sound. We certainly wouldn't mind if anybody from the community publishes it as an npm package, as long as the license is honored. We'd actually appreciate the help :slightly_smiling_face: