Adds flush.file.size to allow file size based rotation

[Usiel]

trafficstars

Problem

See: https://github.com/confluentinc/kafka-connect-hdfs/issues/365

We want to be able to control the size of the files produced by the connector.

Solution

This change gets us in the vicinity of such a feature. A file size guarantee would require a major rewrite, as the current logic is based on partitions.

The file size based rotation is implemented in such a way, that we rotate all open files for a given TopicPartitionWriter instance, when any file has reached the configured limit.

I wanted a solution that does not have to count bytes and do unnecessary serialization, thus I implemented a few wrappers around the existing OutputStreams, which allow us to access the current file size. This implies that, even with small messages, we will produce files that may be larger than configured depending on the size of the output buffers.

I'm happy to work on this more, if you all are open to this feature. For the use case I have at my company, this current version is sufficient.

Does this solution apply anywhere else?

• [ ] yes
• [x] no

If yes, where?

Test Strategy

Testing done:

• [ ] Unit tests (might add some)
• [x] Integration tests
• [ ] System tests
• [x] Manual tests

I am running this on a Kafka Connect 7.2.4 cluster, so far no issues with String and Avro formats.

[cla-assistant[bot]]

All committers have signed the CLA.