Supporting namespaced protos

[bryanhughes]

How do I support generating erlang code for two proto's that have the same name, slightly different structure, and are in different packages:

This file is located in proto/public/foo.proto

syntax = "proto2";

package public;

option cc_enable_arenas = true;
option java_package = "com.example.public";
option java_outer_classname = "FooProto";
option java_multiple_files = true;
option objc_class_prefix = "EDM";


// Other imports
import "google/protobuf/timestamp.proto";

message Foo {
  extensions 100 to 110;
  optional string bar = 1;
  optional string baz = 2;
  optional google.protobuf.Timestamp ignore_me = 3;
  optional int64 version = 4;
  optional int32 foobar = 5;
}

While this proto is defined in proto/test_schema/foo.proto

syntax = "proto2";

package test_schema;

option cc_enable_arenas = true;
option java_package = "com.example.test_schema";
option java_outer_classname = "FooProto";
option java_multiple_files = true;
option objc_class_prefix = "EDM";


// Other imports

message Foo {
  optional string bar = 1;
  optional string baz = 2;
  optional int64 version = 3;
}

This is a completely valid definition. How to I tell gpb to emit the source code into package directories so that foo_pb.erl is not overwritten. My rebar.config is:

{gpb_opts, [{i, "proto"},
            {o_erl, "src"},
            {o_hrl, "include"},
            use_packages,
            {module_name_suffix, "_pb"},
            {strings_as_binaries, true},
            {recursive, true},
            type_specs]}.

[tomas-abrahamsson]

Maybe you could try the use_packages option? I noticed your Foo messages are in different packages. With the use_packages, messages will get prefixed by package, so you will probably need to change some of your code that refer to message names: from 'Foo' to 'public.Foo' or to 'test_schema.Foo'.

(Dead-end-side-note: I was first thinking another way could be the option {rename, {msg_name, {prefix, {by_proto, [{"foo", "public_"}, {"testfoo", "test_"}]}}}} but then I saw both your protos are named "foo.proto", and the prefix by proto feature works on proto file base names only, so this option is probably not useful for you unless you can rename the files.)

Another idea could perhaps be to use different rebar profiles? Something like rebar3 as my_test_proto_profile compile and then it could perhaps be possible to have a different set of gpb_opts for this profile?

[bryanhughes]

I am using use_packages, which correctly generates the code.

As you saw, the issue is that the file foo_pb.er gets overwritten by the last protobuffer that is processed. I can not rename the files since this is all generated code. When you say prepend, would then result in the files being named: src/public_foo_pb.er and src/test_schema_foo_pb.er? If so, this would work.

[bryanhughes]

Unfortunately, this requires me to manually rename everything.

What would be the best solution would be to set an option to prepend the package name: {rename, {msg_name, {prepend, package_name}}}

So ideally it would generate: src/public_foo_pb.erl and src/test_schema_foo_pb.erl

[tomas-abrahamsson]

I see now, sorry, didn't get originally that it was also the proto files that had the same name (but in different directories), and got also now that you were already using use_packages. Apologies.

@lrascao, are there already possibilities in rebar3_gpb_plugin to specify gpb_opts per proto file, or is it one set of gpb_opts for all proto files? Also, if an option would be added to gpb to derive the .erl file name from the package declaration inside the .proto, would you know if the resulting .erl will get compiled? Background for this 2nd question is I noticed the target file name calculations, but maybe it will just think it needs to be regenerated from .proto every time?

[bryanhughes]

Ugh. Sorry for the typo's in my previous relies - and no worries at all. :)

Just to clarify, the goal is to generate src/public_foo_pb.erl and src/test_schema_foo_pb.erl where the package is prepended to the message file name.

[tomas-abrahamsson]

I pushed to a temporary branch, mod-name-from-pkg, support for a new option module_name_from_package that does what I think you are looking for.

The functionality as such is a bit on the border of what should maybe be in the build system instead and I'm a bit hesitant whether to actually merge this or if it would be better solved in the rebar3_gpb_plugin instead. Maybe @lrascao would like to weigh in? My reasons for thinking it may perhaps better be solved in the build system are: (a) there are already general options for prefixing and suffixing to avoid naming collisions and (b) that even though this might solve your particular case, there would still be errors if the other foo.proto was in the same package, so the new option is no general solution.

[lrascao]

yeah, i agree that this should up to the build system although i don't have any good ideas on how to solve (b), @bryanhughes is this a case you feel warrants some kind of specific behaviour?

@tomas-abrahamsson would gpb_compile:list_io also provide the package in use by the proto file?

[tomas-abrahamsson]

I toyed a bit with a proposal to add per-proto gpb_opts overrides: lrascao/rebar3_gpb_plugin#137 so we can now compare it with the mod-name-from-pkg branch to see which solution we like the best.

would gpb_compile:list_io also provide the package in use by the proto file?

yes, on the branch, the gpb_compile:list_io reads the proto names and returns the erl file name with the .proto's package name in it, so the lrascao/rebar3_gpb_plugin#136 comes very handy to get the file names right.

[bryanhughes]

Apologies - been a crazy week and trying to catch up -- ultimately this issue boils down to namespacing by package names, and how fundamentally Erlang does not support any type of namespacing other than prepending the package to the module name (much like gpb does with the atom message name when you specify use_packages). Ultimately, I have two proto's with the same name but in different packages that get clobbered when the code is generated. I am not sure how the build system would solve this problem unless it created an app per package which seems wrong. It seems like the only real solution is an option that lets me (or others who have this problem) to have the generated files globally prepended by the package name.

@lrascao In regards to my requirements, I would prefer an all or nothing since I am also generating code that maps to the .proto. Requiring the tool to have custom exceptions for protos that collide on code generation is not really helpful. Rather, a flag that simply says when using packages, then also prepend the generated file names with the package name.

@tomas-abrahamsson I have not had a chance to pull your branch to test - I will try this tonight.

Really appreciate the help!

[tomas-abrahamsson]

I'm closing this due to inactivity, and will withdraw mod-name-from-pkg branch as well as the corresponding lrascao/rebar3_gpb_plugin#137 PR.

If this is still of interest, feel free to reopen, and give some indication as to what approach would work best, given the previous discussion.