kafka-connect-storage-common icon indicating copy to clipboard operation
kafka-connect-storage-common copied to clipboard

Published 20 hours ago • verified publisher icon confluentinc

support to configure multiple partitioner and encode sequentially

Open daehokimm opened this issue 4 years ago • 1 comments

Hi All :)

TL;DR

When the s3 partition is blunt, athena's query take long time.

In my environment, the s3 sink connector encode partition by FieldPartitioner. As a result, one of s3 partition has so many objects. and it cause long time query when using athena. if s3 sink connector can configure more than one partitioner(like FieldPartitioner + DailyPartitioner), the s3 objects are encoded in detail.

So more than one partitioning is needed. plz support this 🙏

daehokimm avatar May 31 '21 16:05 daehokimm

You can write your own partitionner and use it with the confluent connector

Look to the current partitionners present in the repo

raphaelauv avatar Jan 20 '22 22:01 raphaelauv