kafka-connect-hdfs icon indicating copy to clipboard operation
kafka-connect-hdfs copied to clipboard

Published 20 hours ago • verified publisher icon confluentinc

it will create many small files?

Open katty0924 opened this issue 4 years ago • 1 comments

Hi, everyone , recently i used hdfs sink connector, and i am worried about that if the data is consistenctly input to kafka, will it generate some small files, which are not good for hdfs namenode ?

katty0924 avatar Feb 23 '21 09:02 katty0924

Depends on your output format, flush size, and partitioner. It is possible to generate several GB files, in my experience

OneCricketeer avatar Mar 02 '21 17:03 OneCricketeer