protobuf
protobuf copied to clipboard
golang
protoc-gen-go: support go_tag option to specify custom struct tags
Is it possible to define a message with custom tags? For example defining a json name that isn't just the lowercase field name or adding a bson tag.
I'd like to add that this feature would be extremely useful for validation purposes.
+1
Use case: Using protobuf generated structs along with sqlx package. Would be awesome
Likewise the go library for cloud datastore. Basically, increasing numbers of things make use of struct tags, and without support for them you're left manually duplicating structs? If there are no plans to add this functionality, what is the recommended way to handle it?
I would love this feature! This seems to somehow mitigate the need
https://github.com/favadi/protoc-go-inject-tag
This would be handy, as I need to validate JSON payloads for my REST API portion of the service.
Since golang/protobuf will probably never support this. Here are some solutions.
https://github.com/mwitkow/go-proto-validators is useful for validation on proto structs.
https://github.com/gogo/protobuf can be used in two ways to have custom tags:
-
You can use the moretags extension https://github.com/gogo/protobuf/blob/master/test/tags/tags.proto
-
Or you can use the typedecl extension if you don't want to generate the golang struct. This allows you declare your own golang struct with all the tags you want and more.
I've add a plugin retag for protoc-gen-go.
https://github.com/qianlnk/protobuf/tree/master/protoc-gen-go/retag
protoc --go_out=plugins=grpc+retag:. yourproto.proto
@qianlnk .thanks.I tried your plugin when i use only message then ther is no problem but when the message have sub message like this
message SendData {
message MetaData{
string Name = 1;//`json:"name;omitempty"`
int64 Length = 2;//`json:"length;omitempty"`
}
message Chunck{
bytes Data = 1;//`json:"data;omitempty"`
int64 Position = 2;//`json:"position;omitempty"`
}
}
This throws the error.Any how the retag can support this??? Thanks...
@qianlnk I am having a problem making this work.... does it support proto3?
syntax = "proto3";
package contracts;
import "Common.proto";
//import "google/protobuf/timestamp.proto";
message ZZZ {
string id = 1; // `db:"id"`
string clientId = 2;
string datetime = 3;
}
Converting with:
protoc -I=xyz.eu/contracts/proto --go_out=plugins=grpc+retag:xyz.eu/contracts/ xyz.eu/contracts/proto/*.proto
Result:
type ZZZ struct {
Id string `protobuf:"bytes,1,opt,name=id" json:"id,omitempty"`
ClientId string `protobuf:"bytes,2,opt,name=clientId" json:"clientId,omitempty"`
Datetime string `protobuf:"bytes,5,opt,name=datetime" json:"datetime,omitempty"`
}
The db custom tag on id is missing.... Like everyone using proto, I need this in order to work with (any) orm:) Thanks for your time
@qianlnk thank you for the response. I changed the proto to:
string ID = 1; // db:"id,omitempty"
In above and below example the ` before db and at the end of line are omitted by github... but they are there. Unlike your example, only I am using a non existing tag like (no json or xml). Still no db output:
ID string protobuf:"bytes,1,opt,name=ID" json:"ID,omitempty"
If you have another idea please let me know. I am looking into the plugin code right now. I tested that the code is called with a print statement in de init. So the environment seems ok.
@atamgp have you install my retag plugin?
git clone https://github.com/qianlnk/protobuf.git $GOPATH/src/github.com/golang/protobuf
go install $GOPATH/src/github.com/golang/protobuf/protoc-gen-go
this is my try:
syntax = "proto3";
package contracts;
//import "Common.proto";
//import "google/protobuf/timestamp.proto";
message ZZZ {
string ID = 1; //`db:"id"`
string ClientID = 2; //`db:"client_id"`
string Datetime = 3; //`db:"datetime"`
}
run:
protoc -I/usr/local/include -I. \
-I$GOPATH/src \
--go_out=plugins=grpc+retag:. \
test.proto
and result:
type ZZZ struct {
ID string ` protobuf:"bytes,1,opt,name=ID" json:"ID,omitempty" db:"id"`
ClientID string ` protobuf:"bytes,2,opt,name=ClientID" json:"ClientID,omitempty" db:"client_id"`
Datetime string ` protobuf:"bytes,3,opt,name=Datetime" json:"Datetime,omitempty" db:"datetime"`
}
@qianlnk OK, I have it figured out a bit more :)
message ZZZZ {
string ID = 1; //db:"id"
string clientID = 2; //db:"id"
double amount = 3;
}
Things I learned:
- I have to put the db tag on all fields of a message. Otherwise I get an exception: panic: runtime error: index out of range (panic: runtime error: index out of range -> indexing [1]) For some fields it is not wanted to be mapped by an ORM. Is it possible to make this optional?
- Like you said earlier, field name has to begin with a capital otherwise its ignored. Ofcourse its possible to misuse this aspect to realize optional fields (1) but is less clean.
- Command path used. I have this folder structure: $GOPATH/src/X/Y/Z/zzz.proto I was using a protoc command from src folder which did not work:
Working: When calling from within Z: protoc --go_out=plugins=grpc+retag:. zzz.proto Not working: protoc -I:X/Y/Z --go_out=plugins=grpc+retag:. X/Y/Z/zzz.proto Working: protoc -I:. -I:X/Y/Z --go_out=plugins=grpc+retag:. X/Y/Z/zzz.proto 4) importing from other proto: I have a common.proto and a zzz.proto. zzz is importing an enum and a message from common. working: if a use a working command from point 3) above for 1 file at a time e.g.: protoc -I:. -I:X/Y/Z --go_out=plugins=grpc+retag:. X/Y/Z/zzz.proto protoc -I:. -I:X/Y/Z --go_out=plugins=grpc+retag:. X/Y/Z/common.proto
not working: same commands which are working for 1 file but in order to process both proto files /*.proto instead of a specific proto. E.g.: protoc -I:. -I:X/Y/Z --go_out=plugins=grpc+retag:. X/Y/Z/*.proto
errors: .... is already defined in file ....
It is really nice to process all proto files in 1 time, can you have a look at this?
Last: It is not clean to have all proto files and generated go files in the same folder. In my case Z is actually named proto and Y is contracts. I want to have my generated go files put in folder Y instead of Z. Still looking for a working combination command for this...
Has anyone found a solution for converting between type: string and type: bson.ObjectId as well?
Please support this it is extremely useful and cuts down on needless code bloat.
While this idea is super useful in Go, unfortunately no other language that proto supports can really make use of it. As such, changing/supporting this would be outside of the scope of the official Go protobuf package.
@puellanivis, that is not true. Supporting custom tags would open protobufs to a plethora of possible tooling.
I'm going to re-open this issue, but it does not mean that we're going to support this. I'd like to see more discussion on the feasibility of this. Some thoughts:
- I can see the usefulness of this feature for Go-heavy projects, but protobufs were designed a language-agnostic way to interchange data. Baking more Go-specific details in the proto files seems to be antithetical to the philosophy of protobufs.
- This features makes the assumptions that generated Go protobufs are always structs with fields. What does it mean if we want the ability to generate opaque protobuf types that are interacted via methods. Performance testing has shown that techniques like lazy unmarshaling can bring significant speed benefits, but is only feasible as an opaque type.
- The proper way to support this seems to be using protobuf options, but I don't see any proposal here suggesting a syntax for how that would work.
why not add a plugin like
https://github.com/qianlnk/protobuf/tree/master/protoc-gen-go/retag
This plugin is my original idea, don't take too much effort to attain it.But I believe that many people need it, so I hope more people can join in.
@qianlnk Looked at your retag plugin as one of several alternatives. I don't really like the idea of cloning it into my golang/protobuf folder, especially since This branch is 8 commits ahead, 28 commits behind golang:master. Would you consider making it a standalone plugin instead?
@dsnet See tags.proto which was referred to by @awalterschulze as an example of using the protobuf options approach.
It could perhaps look something like this:
message Enrollment {
uint64 CourseID = 1 [(go_tags) = "gorm:\"unique_index:uid_user_course\""];
uint64 UserID = 2 [(go_tags) = "gorm:\"unique_index:uid_user_course\""];
int32 foo = 3 [(go_tags) = "sql:\"type:text\""];
string bar = 4;
}
This would be a very useful addition to avoid a lot of boilerplate, and IMO it would fit nicely in with other language-specific options, such as e.g. java_package to name just one. Just like java_package is ignored when generating Go files, go_tags would be ignored when generating non-Go code.
The only issue that I don't like with the above is the need for the \". It would be nice if that can be avoided, but it is a compromise I don't mind if we can get this feature into the main project.
@dsnet Regarding your point about opaque types. I think there are two main use cases that have come up in this issue.
- Boilerplate avoidance for database/storage backends.
- Validation of input fields.
I think the first is best dealt with using something like my proposed go_tags, because the tools that process the tags are their own beasts and it seems difficult to interact via lazy functions.
Validation of input fields seems different though. I can imagine that validation could be done as part of lazy unmarshaling, but that would require another option such as in go-proto-validators. However, such lazy unmarshaling with validation is not Go-specific, and so I don't think it makes sense to put such validation data into struct tags in the generated code. Indeed, go-proto-validators does not put validation data in struct tags.
IMO, it's better to provide a global option, for example go_tags with value like "bson" or "bson,xml".
It could perhaps look something like this:
syntax="proto3";
package user;
option go_package = "user";
option go_tags = "bson";
message User {
string id = 1;
string name = 2;
}
Which generates:
type User struct {
Id string `protobuf:"varint,1,opt,name=id" json:"id,omitempty" bson:"id,omitempty"`
Name string `protobuf:"varint,1,opt,name=name" json:"name,omitempty" bson:"name,omitempty"`
}
protoc-gen-go generates json by default, but sometime we need others tags for marshal/unmarshal. As a result, I have to write another struct the same as generated except tags.
A global option may work for your use case with bson, but many other use cases require per field options.
@meling I don't thinking it's a good idea to insert more info to every field option. protoc-gen-go bind struct with json, but we use struct in many cases, such as bson and xml and some custom cases. The global option mentioned above is to solve the problem, which is not only in my case.
To database/storage, for example gorm, use tags to define index/unique_index is a crazy action. We use protobuf with many languages, option for every field is useless for python/java. These options make the .proto unreadable for python/java developers. What's more, option for every field makes the .proto file hard to read and modify, for there are many service logics. Simple is better.
Why is it crazy to define index fields using standard protobuf options syntax? I didn’t design the option syntax for protobuf, but I learn to adapt. Just because something is unreadable to some people doesn’t mean it is crazy. What is crazy is to do “manual” conversion between a protobuf struct and a database struct when they are otherwise identical, because that is just error prone... and slow.
Of course your approach is simple, but it doesn’t solve the problems of at least half of the people asking for this feature...