rock icon indicating copy to clipboard operation
rock copied to clipboard

Update Kafka to add more partitions

Open dcode opened this issue 5 years ago • 0 comments

Kafka scales by adding partitions. By default, we currently only create a single partition per topic. The number of partitions should be a multiple of the number of consumers. In the rocknsm stack alone, that's equivalent to the number of logstash consumer threads per topic which defaults to 1. We should increase the number of partitions to maybe a dozen by default. It doesn't hurt anything and only provides the capability to scale as needed.

dcode avatar Apr 14 '19 17:04 dcode