events distribution in kafka partitions

[galactus009]

Hello

We are having unequal events distribution of events in the partitions. our topic have 8 partitions we are noticing that one of the partition getting bulk of the events others paritions are getting very less. please advise.

Below is our fluentd-kafka-output config

<store>
	@type kafka_buffered
   	 brokers broker1:9092,broker2:9092,broker3:9092 
       default_topic MyTopic
       output_data_type msgpack
       output_include_tag true
       get_kafka_client_log false
       buffer_type file
       buffer_path /tmp/td-agent/buffer
       flush_interval 3s
       required_acks 1
       num_threads 8
</store>

[galactus009]

Any guidance is appreciated

[repeatedly]

https://github.com/zendesk/ruby-kafka/blob/master/lib/kafka/partitioner.rb

Here is an implementation of partitioner. I'm not sure why rand returns wrong distribution. One approach is specify partition_key or partition_key_key to use value based distribution.

[punneng]

It somehow parses message_key and partition_key as empty string which supposed to be nil Since message_key and partition_key are empty string then it doesnt go through rand(partition_count) condition.

im investigating and update or open a PR after I figure out why setting

default_partition_key "#{ENV['FLUENT_KAFKA_DEFAULT_PARTITION_KEY'] || nil}"
default_message_key "#{ENV['FLUENT_KAFKA_DEFAULT_MESSAGE_KEY'] || nil}"

parses default_partition_key and default_message_key as empty string.

hope it helps.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days