kafka-connect-protobuf-converter
kafka-connect-protobuf-converter copied to clipboard
Fields with same name and different type can cause issue for avro sink
Assume a following simple protobuf schema
message NestedStruct_1 {
optional int64 nestedField_1
}
message NestedStruct_2 {
optional int64 nestedField_2
}
message Struct_1 {
optional NestedStruct_1 nested = 1;
}
message Struct_2 {
optional NestedStruct_2 nested = 1;
}
message Test {
required Struct_1 struct1 = 1;
required Struct_2 struct2 = 2;
}
Now, protobuf converter can read the data for Test.proto but if you try to write the this data in avro, sink, you will get an exception similar to follows
org.apache.avro.SchemaParseException: Can't redefine: nested
In the original schema, there are two fields with same name but different type. However, when building connect schema in ProtobufData, we lose the type scope and hence it results in conflicts in Avro sinks.
If we can include the type name in the field, the conflict can be avoided. I can fix this if this solution sounds alright. Other suggestions are welcome as well.