rock
rock copied to clipboard
Update Kafka to add more partitions
Kafka scales by adding partitions. By default, we currently only create a single partition per topic. The number of partitions should be a multiple of the number of consumers. In the rocknsm stack alone, that's equivalent to the number of logstash consumer threads per topic which defaults to 1. We should increase the number of partitions to maybe a dozen by default. It doesn't hurt anything and only provides the capability to scale as needed.