How to get protoc-gen-go-drpc

[cindyzqtnew]

Hi folks,

I'm a bit new to the proto and trying to replace the existing grpc with drpc in my project.

I look into the doc and it says "Place protoc-gen-go-drpc in your $PATH and have protoc generate your definitions", but find nowhere to get the binary protoc-gen-go-drpc. There is only a simple main.go and useless README under storj.io/drpc/cmd/protoc-gen-go-drpc.

May I know how to generate the drpc go code based on the proto I already have and where to get the required tool?

Appreciate for your help!

[egonelbre]

This should do the trick:

go get storj.io/drpc/cmd/protoc-gen-go-drpc

The quick-start guide https://storj.github.io/drpc/docs.html does contain the same command below the text you quoted.

[cindyzqtnew]

This should do the trick:

go get storj.io/drpc/cmd/protoc-gen-go-drpc

The quick-start guide https://storj.github.io/drpc/docs.html does contain the same command below the text you quoted.

@egonelbre I've tried it but it always return the not found error.

go get storj.io/drpc/cmd/protoc-gen-go-drpc
# cd .; git clone -- https://github.com/storj/drpc/cmd/protoc-gen-go-drpc /go_path/src/storj.io/drpc/cmd/protoc-gen-go-drpc
Cloning into '/go_path/src/storj.io/drpc/cmd/protoc-gen-go-drpc'...
remote: Not Found
fatal: repository 'https://github.com/storj/drpc/cmd/protoc-gen-go-drpc/' not found
package storj.io/drpc/cmd/protoc-gen-go-drpc: exit status 128

[egonelbre]

@cindyzqtnew which version of Go are you using?

Maybe this will work better?

go install storj.io/drpc/cmd/protoc-gen-go-drpc@latest

[cindyzqtnew]

go install storj.io/drpc/cmd/protoc-gen-go-drpc@latest

@egonelbre I'm using go1.15. It seems "@version" is not supported in this version.

[egonelbre]

So, I debugged a little. It seems the redirection isn't working quite right for go1.15 for some reason. It'll take time to make and deploy the fix.

If you can then update your Go to latest version... Using go install does seem to work correctly.

If updating Go is a problem then you can use this workaround to use a newer Go version:

go get golang.org/dl/go1.17.6
go1.17.6 download
go1.17.6 install storj.io/drpc/cmd/protoc-gen-go-drpc@latest

[cindyzqtnew]

So, I debugged a little. It seems the redirection isn't working quite right for go1.15 for some reason. It'll take time to make and deploy the fix.

If you can then update your Go to latest version... Using go install does seem to work correctly.

If updating Go is a problem then you can use this workaround to use a newer Go version:

go get golang.org/dl/go1.17.6
go1.17.6 download
go1.17.6 install storj.io/drpc/cmd/protoc-gen-go-drpc@latest

@egonelbre thanks for your help! I'm able to download and install the protoc-gen-go-drpc by the workaround. My project can not be easily upgraded to 1.17. Then I tried to generate the code as follows but it seems not using the drpc protoc?

> /go_path/native/x86_64-pc-linux-gnu/protobuf/protoc --go_out=. --go_opt=paths=source_relative --go-drpc_out=. --go-drpc_opt=paths=source_relative dheartbeat.proto
protoc-gen-go: program not found or is not executable
--go_out: protoc-gen-go: Plugin failed with status code 1

BTW, gogoproto like "goproto_stringer" or "gogoproto.nullable" is widely used in my project. Are they adapted in dRPC too?

[egonelbre]

The --go_out options imply that you also want to use go protobuf generator:

go install google.golang.org/protobuf/cmd/protoc-gen-go@latest

If you want to use gogoproto, then you'll need the gogo generators and specify those arguments as usual. See https://github.com/gogo/protobuf#installation for details.

But, yes, dRPC should work fine with gogoproto.

[cindyzqtnew]

The --go_out options imply that you also want to use go protobuf generator:

go install google.golang.org/protobuf/cmd/protoc-gen-go@latest

If you want to use gogoproto, then you'll need the gogo generators and specify those arguments as usual. See https://github.com/gogo/protobuf#installation for details.

But, yes, dRPC should work fine with gogoproto.

@egonelbre e I'm able to generate the drpc code by the proto file. Hooray~

I still have some questions.

1. In generated code, it uses "google.golang.org/protobuf/proto". Is it ok to use "github.com/gogo/protobuf/proto"?
2. If I don't use JSON, I guess I can delete the JSON Marshal/UnMarshal methods, right?
3. Currently in my project with grpc, I use Marshal/Unmarshal only. What's the difference between Marshal and MarshalAppend?

[egonelbre]

1. The generated code depends on whether you use the official proto or gogo proto. It should pick the right one and you shouldn't need to modify the generated code.
2. Yes, however, usually it's easier to ignore those.
3. That's a question for protobuf team -- however, the MarshalAppend allows to reuse an existing byte buffer and hence reduce allocations.

[cindyzqtnew]

@egonelbre One more question. In the doc example, service is registered by the drpcmutex and before drpcserver is created. Is it ok to register more services by the drpcmutex after drpcserver creation. After all, services should be registered before the drpcserver serve.

[egonelbre]

Ah, to clarify the gogoproto thing, I missed that it also needs a few additional arguments:

protoc --gogo_out=paths=source_relative:. --go-drpc_out=paths=source_relative,protolib=github.com/gogo/protobuf:. service.proto

[egonelbre]

At the moment it wouldn't be correct to register after Serve, otherwise it should be fine. If you run your program with -race it should point out the problems.

[cindyzqtnew]

Ah, to clarify the gogoproto thing, I missed that it also needs a few additional arguments:

protoc --gogo_out=paths=source_relative:. --go-drpc_out=paths=source_relative,protolib=github.com/gogo/protobuf:. service.proto

@egonelbre I see. So whether using official proto or gogo proto is specified by the option in protoc?

[cindyzqtnew]

At the moment it wouldn't be correct to register after Serve, otherwise it should be fine. If you run your program with -race it should point out the problems.

@egonelbre got you. I'll try to generate all code from grpc to drpc in my project (only change 1 proto file for testing at the moment) and then test for the performance difference.

Again thanks a lot for your help.

[cindyzqtnew]

@egonelbre It seems adding argument "--go-drpc_out=paths=source_relative,protolib=github.com/gogo/protobuf/proto" makes the generated code not working correctly.

mport (
	context "context"
	errors "errors"
	proto "github.com/gogo/protobuf/proto"
	drpc "storj.io/drpc"
	drpcerr "storj.io/drpc/drpcerr"
)

type drpcEncoding_File_dheartbeat_proto struct{}

func (drpcEncoding_File_dheartbeat_proto) Marshal(msg drpc.Message) ([]byte, error) {
	return proto.Marshal(msg)
}

It shows the error that "msg missing method". I need to manually change it to "msg.(proto.Message)".

What's more, is there any argument to control not generating the JSON related code? The reason is "github.com/gogo/protobuf/proto" does not have "JSONMarshal" / "JSONUnmarshal" and in the end I will need to update the Makefile to automatically generate the drpc code.

[egonelbre]

@cindyzqtnew did you also use gogo for rest of the generation and rest of the codebase? Are you also using a protobuf version that is compatible with gogo?

[cindyzqtnew]

@cindyzqtnew did you also use gogo for rest of the generation and rest of the codebase? Are you also using a protobuf version that is compatible with gogo?

@egonelbre here is the generate cmd I use.

> /go_path/native/x86_64-pc-linux-gnu/protobuf/protoc --go-drpc_out=. --go-drpc_opt=paths=source_relative,protolib=github.com/gogo/protobuf/proto dheartbeat.proto

[egonelbre]

@cindyzqtnew that part does not contain the code to generate the protobuf messages, it contains only the service parts. The problem is in the --gogo_out argument.

[cindyzqtnew]

@cindyzqtnew that part does not contain the code to generate the protobuf messages, it contains only the service parts. The problem is in the --gogo_out argument.

@egonelbre it seems not working and wired..

> /go_path/native/x86_64-pc-linux-gnu/protobuf/protoc --gogo_out=paths=source_relative --go-drpc_out=. --go-drpc_opt=paths=source_relative,protolib=github.com/gogo/protobuf/proto dheartbeat.proto
paths=source_relative/: No such file or directory

[egonelbre]

@cindyzqtnew the command you pasted does not match https://github.com/storj/drpc/issues/23#issuecomment-1016181329

[cindyzqtnew]

@egonelbre

Now I'm able to generate the code manually by go1.17.6 install storj.io/drpc/cmd/protoc-gen-go-drpc@latest. Since my project is big, I already automate the generate work in the Makefile. In the Makefile, I need to install the protoc-gen-go-drpc plugin first. I usually place the file under pkg/cmd/ and install it by go install. Which files should I download for installation? The whole repo of "https://github.com/storj/drpc"? But I don't see there is any main there.

[cindyzqtnew]

@egonelbre

Now I'm able to generate the code manually by go1.17.6 install storj.io/drpc/cmd/protoc-gen-go-drpc@latest. Since my project is big, I already automate the generate work in the Makefile. In the Makefile, I need to install the protoc-gen-go-drpc plugin first. I usually place the file under pkg/cmd/ and install it by go install. Which files should I download for installation? The whole repo of "https://github.com/storj/drpc"? But I don't see there is any main there.

@egonelbre

BTW, here is how I generate the gRPC in the Makefile.

bins:
...
    bin/protoc-gen-gogo
...

bin/grpc_proto_src: $(PROTOC) $(GO_PROTOS) $(GOGOPROTO_PROTO) bin/.bootstrap bin/protoc-gen-gogo
...
for dir in $(sort $(dir $(GO_PROTOS))); do \
	  $(PROTOC) -Ipkg:./vendor/github.com:$(GOGO_PROTOBUF_PATH):$(PROTOBUF_PATH) --gogo_out=$(PROTO_MAPPINGS),plugins=grpc,import_prefix=test/pkg/:./pkg $$dir/*.proto; \
	done
...

I try to install drpc and generate the code as follows.

bin/protoc-gen-go-drpc: bin/.submodules-initialized
	cd pkg/cmd/protoc-gen-go-drpc && GO111MODULE=on $(GO_INSTALL) -v ./ && cd -

bin/drpc_proto_src: $(PROTOC) $(GO_PROTOS) $(GOGOPROTO_PROTO) bin/.bootstrap bin/protoc-gen-go-drpc
...
for dir in $(sort $(dir $(GO_PROTOS))); do \
	  $(PROTOC) -Ipkg:./vendor/github.com:$(GOGO_PROTOBUF_PATH):$(PROTOBUF_PATH) --go-drpc_out=paths=source_relative,import_prefix=test/pkg/:./pkg $$dir/*.proto; \
	done
...

It seems not working...

Appreciate for any help!

[zeebo]

It's unclear what wasn't working about your Makefile. I hope you figured it out!