kafka-connect-storage-common icon indicating copy to clipboard operation
kafka-connect-storage-common copied to clipboard

Conflict between partition fields and the data source

Open nutteen opened this issue 6 years ago • 0 comments

We are using DailyPartitioner for our HDFS sink connector with hive integration. The source of a topic is from Debezium that captures the change from a source table that contains fields called month, year.

This leads to the situation where we have ambiguous fields (two year columns in hive table - one creates from the partitioner and another one from the source).

Is there any way to workaround this at the moment? Shouldn't we use the prefix for the generated partition fields to avoid the conflict? e.g. Use _year, _month, _day for DailyPartitioner

nutteen avatar May 24 '19 03:05 nutteen