grpc-tools grpc output references protobuf files that were not created

[eighty4]

Problem description

I encountered a head scratcher (maybe not a bug) my first time trying to use grpc-node's packages.

foo_grpc_pb.js seems to reference pb models at foo_pb.js, except the models are each created in their own js module, without a comprehensive _pb.js.

Reproduction steps

With the package.json and reservations.proto files below, I run the command npm run protoc, which is: grpc_tools_node_protoc ./reservations.proto --proto_path=. --js_out=. --grpc_out=.

I would expect to see output for protobuf models and the grpc service that references the generated protobuf sources. The file reservations_grpc_pb.js references the file reservations_pb.js with a require. However, I only see reservationopresult.js, reservationidentifier.js and reservation.js files created for protobuf.

If the command is correct but the output is incorrect, this should probably be handled as an error state. Am I missing a config option? I see generate_package_definition and grpc_js in grpc-node/packages/grpc-tools/src/node_plugin.cc that I didn't encounter in the docs or guides.

./package.json

{
  "name": "learning-protobuf-and-grpc-js",
  "scripts": {
    "protoc": "grpc_tools_node_protoc ./reservations.proto --proto_path=. --js_out=. --grpc_out=."
  },
  "dependencies": {
    "@grpc/grpc-js": "^1.8.0"
  },
  "devDependencies": {
    "grpc-tools": "^1.12.3"
  }
}

./reservations.proto

syntax = "proto3";

service Reservations {
  rpc CreateReservation(Reservation) returns (ReservationOpResult) {}
}

message Reservation {
  string when = 1;
  uint32 id = 1;
}

message ReservationOpResult {
  bool success = 1;
  optional string error = 2;
}

Environment

• OS name, version and architecture: OSX 13.0.1
• Node version 18.12.1
• Node installation method: Installer from Nodejs.org's downloads page
• If applicable, compiler version: N/a
• Package name and version: grpc-tools 1.12.3

Additional context

[eighty4]

I spoke too soon. I found the grpc_js and generate_package_definition notes in the readme and tried them and neither of them had an impact on my issue.

[eighty4]

The README.md in grpc/grpc's node example configures the protoc command correctly for the output I expected:

grpc_tools_node_protoc --js_out=import_style=commonjs,binary:../node/static_codegen/ --grpc_out=grpc_js:../node/static_codegen helloworld.proto

The repositories grpc/grpc, grpc/grpc-node, protocolbuffers/protobuf and protobufjs/protobuf.js don't have any references to import_style or binary options. I ran my protoc command with just --js_out=binary:grpc_gen and the output diff showed the results were identical. Any clarity on what that is?

[murgatroid99]

I have never seen that grpc_gen before. Where did you get it from?

I would like to note that Protobuf.js is a separate project that is unrelated to the grpc-tools package.

[mattzucker]

I'm having this same problem. I get *_grpc_pb.js files along with a bunch of .js, but it does not generate *_pb.js. The grpc_pb.js references _pb.js but they are no where to be found. Please help!

[murgatroid99]

@eighty4 seemed to indicate that using the options provided in the example they linked generated the output they expected. @mattzucker did you use those same options?

[mattzucker]

I ran this command

grpc_tools_node_protoc --js_out=import_style=commonjs,binary:./dist/js/ --grpc_out=grpc_js:./dist/js/ --proto_path=./proto ./proto/*.proto

[murgatroid99]

OK, can you share a repo with the same directory structure that you are working with, including the .proto files, and a script that produces the output you are seeing, so that I can see exactly what is happening?

[mattzucker]

https://github.com/mattzucker/grpc-test

[eighty4]

The default output is for use within Google's ecosystem. Add the import_style option to produce something usable outside of Google.

--js_out=import_style=commonjs,binary:../node/static_codegen/ 

The binary opt is documented in the example, but it is a noop with diffs and neither option exists in any of the codebases that I could find.

[eighty4]

Further feedback: --grpc_out isn't documented by protoc's help output, neither does protoc --help show the options of included plugins, like --js_out and I'm assuming --go_out

[mattzucker]

Got it working. I was already using the import_style option, but because of your comment regarding binary as a noop I decided to see what would happen if I took that out and only ran it with commonjs. Magically it creates only the 2 files I expected.

Final command here that works for anyone else that comes across this:

grpc_tools_node_protoc --js_out=import_style=commonjs:./dist/js/ --grpc_out=grpc_js:./dist/js/ --proto_path=./proto ./proto/*.proto

[mattzucker]

I was wrong. It has to do with the comma in the command and the fact that I'm running it on Windows with Powershell. Incredibly frustrating. Thank you @eighty4 for helping me debug.

grpc_tools_node_protoc --js_out=import_style="commonjs,binary":./dist/js/ --grpc_out=grpc_js:./dist/js/ --proto_path=./proto ./proto/*.proto

[eighty4]

Ah, I didn't see that.

I wonder why golang (a Google born language) and Node (built on a Google library) aren't officially supported grpc/protobuf languages.

[hubster1]

You can escape special characters with backtick. https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_parsing?view=powershell-7.4#handling-special-characters

grpc_tools_node_protoc --js_out=import_style=commonjs`,binary:./gen/ --grpc_out=grpc_js:./gen/ myservice.proto