kafka-connect-protobuf-converter icon indicating copy to clipboard operation
kafka-connect-protobuf-converter copied to clipboard

Fields with same name and different type can cause issue for avro sink

Open abhishekagarwal87 opened this issue 5 years ago • 1 comments

Assume a following simple protobuf schema

message NestedStruct_1 {
 optional int64 nestedField_1
}

message NestedStruct_2 {
 optional int64 nestedField_2
}

message Struct_1 {
 optional NestedStruct_1 nested = 1;
}

message Struct_2 {
 optional NestedStruct_2 nested = 1;
}

message Test {
   required Struct_1 struct1 = 1;
   required Struct_2 struct2 = 2;
}

Now, protobuf converter can read the data for Test.proto but if you try to write the this data in avro, sink, you will get an exception similar to follows org.apache.avro.SchemaParseException: Can't redefine: nested

In the original schema, there are two fields with same name but different type. However, when building connect schema in ProtobufData, we lose the type scope and hence it results in conflicts in Avro sinks.

abhishekagarwal87 avatar Jan 13 '20 12:01 abhishekagarwal87

If we can include the type name in the field, the conflict can be avoided. I can fix this if this solution sounds alright. Other suggestions are welcome as well.

abhishekagarwal87 avatar Jan 13 '20 12:01 abhishekagarwal87