Configuring Jitsu Bulker for Multi-Partition Kafka Topics

[ZiyaadQasem]

In Kubernetes deployment of Jitsu, the Bulker component is responsible for batching events and sending them to a ClickHouse instance. Currently, the Kafka topic that Bulker creates and consumes is configured with only one partition.

Challenges Encountered:

• Partition Limitation: The single-partition setup leads to performance bottlenecks and limits scalability.
• Data Rebalancing Issues: Attempting to manually increase the number of partitions on the existing Kafka topic results in data rebalancing problems, which can disrupt the data flow and processing.

Questions:

• How can I instruct Jitsu Bulker to create Kafka topics with multiple partitions during their initial creation?

• Are there specific configuration settings or parameters within Jitsu or Bulker that allow specifying the desired number of partitions for Kafka topics?