examples icon indicating copy to clipboard operation
examples copied to clipboard

Consider adding Storage Connectors

Open OneCricketeer opened this issue 5 years ago • 2 comments

I see there are examples for JDBC and Elastic, so perhaps the storage based connectors can be included as well?

HDFS: A Hadoop installation shouldn't be required since HDFS URI can be set to local filesystem. I've tested with hdfs.url=file:///tmp/kafka-connect, and it worked as expected.

S3: An AWS account isn't required. See - https://blog.minio.io/journaling-kafka-messages-with-s3-connector-and-minio-83651a51045d

I have some workable code for at least S3 here - https://github.com/cricket007/kafka-connect-sandbox


Extras: Use something like Apache Drill to connect to the local Zookeeper Server and run Drill SQL queries over data (can query JSON and Avro files)

OneCricketeer avatar Jul 26 '18 05:07 OneCricketeer

@cricket007 : thanks for the suggestion, we'll look into it.

ybyzek avatar Jul 26 '18 15:07 ybyzek

Minio example: https://github.com/framiere/a-kafka-story/tree/master/step13

ybyzek avatar Aug 22 '18 14:08 ybyzek