The directory and filename customization on HDFS

[zallan114]

Hi, I made a study on this project, for directory and filename stored in HDFS, I got 2 points, can any contributor give a confirmation??

1. directory - directory can be customized by overriding/extending DefaultPartitioner(but topic name can not be overrided, or it may cause offset resetting wrongly, hardcoded in https://github.com/confluentinc/kafka-connect-hdfs/blob/master/src/main/java/io/confluent/connect/hdfs/FileUtils.java#L103).

2. filename - this is hardcoded, currently have no way for extension or overriding, am I wrong here, pls help: https://github.com/confluentinc/kafka-connect-hdfs/blob/master/src/main/java/io/confluent/connect/hdfs/FileUtils.java#L77

[zallan114]

for the first question, I have tested without issue, so self-confirmed.

for the second filename customization, anyone can help, no way to make extention or change??

[zallan114]

I can see some info in confluent codes:

("HDFS connector does not commit consumer offsets to Kafka. Upon startup, HDFS " + "Connector restores offsets from filenames in HDFS. In the absence of files in HDFS, " + "the connector will attempt to find offsets for its consumer group in the " + "'__consumer_offsets' topic. If offsets are not found, the consumer will " + "rely on the reset policy specified in the 'consumer.auto.offset.reset' property to " + "start exporting data to HDFS."

does this mean we can not customize the filename except we totally re-write this part in this project which is too expansible....

[noake]

hi, I find when integrate with hive, it map topic name to tableName in method io.confluent.connect.hdfs.DataWriter.syncWithHive , Can I modify topic name in a transformation to match a valid tableName?

sorry for my poor English. :)