gripmock
gripmock copied to clipboard
tokopedia
Absolute imports for proto files in the same directory not working
Absolute imports for proto files in the same directory not working
In case two or more proto files are in the same package and there is a dependency between them, the go code generation will not work correctly in case the import statement is provided in absolute path. It works correctly if the import statement is relative.
Example:
In a project with the following directory structure:
❯ tree
.
├── proto
│ └── account
│ ├── models.proto
│ └── services.proto
└── readme.md
where the files have the following content:
-
models.proto
syntax = "proto3";
package services;
option go_package = "github.com/proto/services";
message Balance {
int64 units = 1;
string currency = 2;
}
-
services.proto
syntax = "proto3";
package services;
option go_package = "github.com/proto/services";
import "account/models.proto";
service BalanceSvc {
rpc GetBalance(GetBalanceRequest) returns (GetBalanceResponse);
}
message GetBalanceRequest {
int64 account_id = 1;
}
message GetBalanceResponse {
Balance balance = 1;
}
In this example we have a services.proto which defines the grpc services and a models.proto which defines the models for the service. services.proto imports models.proto and both are in the same package and define same go_package. Notice how the --imports flag allows gripmock to find the models.proto but it seems like it is found twice and therefore issues an error. The command below is issued from the project root:
❯ docker run -p 4770:4770 -p 4771:4771 -v "$PWD/proto:/proto" tkpd/gripmock --imports="/protobuf,/proto" /proto/account/services.proto /proto/account/models.proto
Starting GripMock
Serving stub admin on http://:4771
models.proto:10:9: "services.Balance.units" is already defined in file "account/models.proto".
models.proto:11:10: "services.Balance.currency" is already defined in file "account/models.proto".
models.proto:9:9: "services.Balance" is already defined in file "account/models.proto".
2022/06/20 06:19:22 Fail on protoc exit status 1
Interestingly enough, changing the import statement in services.proto from import "account/models.proto"; to import "models.proto"; works.
Attached you can find a zip file containing the project to reproduce the issue. absolute-import-path-bug.zip
when you're not using gripmock, how you pass the argument into protoc? since above error is directly from protoc.
I have similar issue. Service references shared models (shared between multiple services) and I add them as to imports. Doesn't matter if they are in the same folder (/protos) or multiple import paths (/protos,/common), gripmock seems to find them and even read package name but fails to load them, see sample output (just replaced companu name with XXX)
grpc server pid: 65
/go/src/grpc/server.go:19:2: cannot find package "XXX/commonprotos/rpc/EmptyResponse" in any of:
/usr/local/go/src/XXX/commonprotos/rpc/EmptyResponse (from $GOROOT)
/go/src/XXX/commonprotos/rpc/EmptyResponse (from $GOPATH)
2022/07/19 14:24:49 exit status 1
With v1.11-beta message is a bit different (tries to load definition from web) but seems the problem is the same:
2022/07/19 14:25:35 error on fixGoPackage exit status 1
go: finding module for package XXX/commonprotos/rpc/EmptyResponse
/go/src/grpc/server.go:19:2: cannot find module providing package XXX/commonprotos/rpc/EmptyResponse: unrecognized import path "XXX/commonprotos/rpc/EmptyResponse": reading https://XXX/commonprotos/rpc/EmptyResponse?go-get=1: 404 Not Found
2022/07/19 14:25:38 exit status 1
And if I try to add common proto to list of protos explicitly (i.e. '/common/XXX/api/rpc/empty_response.proto' '/proto/AccountService.proto'), it fails like in author's case that type is already defined:
2022/07/19 15:24:39 error on fixGoPackage exit status 1
XXX/api/rpc/empty_response.proto:16:39: "XXX.API.CommonProtos.RPC.EmptyResponse.status" is already defined in file "empty_response.proto".
XXX/api/rpc/empty_response.proto:15:9: "XXX.API.CommonProtos.RPC.EmptyResponse" is already defined in file "empty_response.proto".
AccountService.proto:4:1: Import "XXX/api/rpc/empty_response.proto" was not found or had errors.
AccountService.proto:53:73: "XXX.API.CommonProtos.RPC.EmptyResponse" seems to be defined in "empty_response.proto", which is not imported by "AccountService.proto". To use it here, please add the necessary import.
AccountService.proto:56:69: "XXX.API.CommonProtos.RPC.EmptyResponse" seems to be defined in "empty_response.proto", which is not imported by "AccountService.proto". To use it here, please add the necessary import.
2022/07/19 15:24:39 Fail on protoc exit status 1
To generate real service code we use dotnet Grpc.Tools, which also uses protoc under the hood, and from logs I see it simly uses multiple --proto_path directives w/o any issue, so it seems like some Go specific issue, but I'm not sure how to diagnose :)
Here is simplified example of the issue (see code attached test.zip) Something simple that uses well known google type, works fine.
version: "3.7"
services:
hello-service:
image: tkpd/gripmock:latest
ports:
- '4770:4770'
- '4771:4771'
command:
- '--imports=/protobuf,/proto'
- '/proto/hello-service.proto'
volumes:
- './../TestCompany.Service/proto:/proto:ro'
Starting GripMock
Serving stub admin on http://:4771
grpc server pid: 60
Serving gRPC on tcp://:4770
If using some imports from external folder, getting error:
version: "3.7"
services:
job-service:
image: tkpd/gripmock:latest
ports:
- '4770:4770'
- '4771:4771'
command:
- '--imports=/protobuf,/common,/proto'
- '/proto/job-service.proto'
volumes:
- './../TestCompany.Common/proto:/common:ro'
- './../TestCompany.Service/proto:/proto:ro'
Starting GripMock
Serving stub admin on http://:4771
grpc server pid: 65
/go/src/test.company/services/test-service/job-service.pb.go:19:2: cannot find package "test.company/common/rpc/status" in any of:
/usr/local/go/src/test.company/common/rpc/status (from $GOROOT)
/go/src/test.company/common/rpc/status (from $GOPATH)
2022/07/19 18:31:17 exit status 1
And to reproduce author's behavior, add dependency to compile explicitly and it fails with duplicated definitions:
version: "3.7"
services:
job-service:
image: tkpd/gripmock:latest
ports:
- '4770:4770'
- '4771:4771'
command:
- '--imports=/protobuf,/common,/proto'
- '/common/tc/rpc/status.proto'
- '/proto/job-service.proto'
volumes:
- './../TestCompany.Common/proto:/common:ro'
- './../TestCompany.Service/proto:/proto:ro'
Starting GripMock
Serving stub admin on http://:4771
tc/rpc/status.proto:11:31: "TestCompany.Common.Rpc.Status.code" is already defined in file "status.proto".
tc/rpc/status.proto:12:10: "TestCompany.Common.Rpc.Status.message" is already defined in file "status.proto".
tc/rpc/status.proto:13:32: "TestCompany.Common.Rpc.Status.details" is already defined in file "status.proto".
tc/rpc/status.proto:14:10: "TestCompany.Common.Rpc.Status.trace_id" is already defined in file "status.proto".
tc/rpc/status.proto:10:9: "TestCompany.Common.Rpc.Status" is already defined in file "status.proto".
job-service.proto:4:1: Import "tc/rpc/status.proto" was not found or had errors.
job-service.proto:19:3: "TestCompany.Common.Rpc.Status" seems to be defined in "status.proto", which is not imported by "job-service.proto". To use it here, please add the necessary import.
2022/07/20 06:38:15 Fail on protoc exit status 1
+1 , we also have same issue. As a temp solution , we had to put all the required protos under the same folder and updated the imports accordingly.
I solved the issue by specifying all imported files explicitly along the main proto file. It would be helpful if gripmock could accept path pattern (glob) and expand it automatically like:
docker run -p 4770:4770 -p 4771:4771 -v c:\temp\prototest:/proto tkpd/gripmock /proto/**/*.proto
https://stackoverflow.com/questions/26809484/how-to-use-double-star-glob-in-go
