kafka_group doesn't react on adding partitions to a topic

[mfvanek]

trafficstars

Hi there.

When we add partitions to a topic, fluent-plugin-kafka doesn't detect these changes and doesn't start consuming messages from new partitions. We use kafka_group. As the workaround we just restart our fluentd instances. It is enough to restart at least one instance.

Is it expected behavior?

[mfvanek]

Repro steps We send logs to Kafka, consumes them with Fluentd and persist them into Elastic.

Fluentd configuration

<source>
 @type kafka_group
 brokers broker:9092
 consumer_group my-consumer-group
 topics my-topic
 format json
</source>

<match **>
  @type stdout
</match>

1. Create a topic with one partition

kafka-topics \
--create \
--bootstrap-server broker:9092 \
--replication-factor 1 \
--partitions 1 \
--topic my-topic

2. Run fluetnd

3. Send logs to Kafka:

kafka-console-producer --broker-list broker:9092 --topic my-topic
>1
>2
>3
>4

Logs:

2020-10-19 14:08:57.042342952 +0000 my-topic: 1
2020-10-19 14:08:57.042384921 +0000 my-topic: 2
2020-10-19 14:08:57.042392425 +0000 my-topic: 3
2020-10-19 14:09:00.070712536 +0000 my-topic: 4

4. Visible list of partitions in consumer-groups

kafka-consumer-groups --bootstrap-server broker:9092 --group my-consumer-group --describe
GROUP              TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                HOST            CLIENT-ID
my-consumer-group  my-topic        0          -               0               -               kafka-975c4e8e-ae7a-437c-988e-c740923e5834 /10.116.1.147   kafka

5. Increase partitions count up to 2:

kafka-topics --zookeeper zookeeper:2181 \
--alter \
--topic my-topic \
--partitions 2

6. Send logs to Kafka:

kafka-console-producer --broker-list broker:9092 --topic my-topic
>1
>2
>3
>4

Logs:

2020-10-19 14:09:57.132099077 +0000 my-topic: 1
2020-10-19 14:09:57.132111450 +0000 my-topic: 2
2020-10-19 14:09:57.132118563 +0000 my-topic: 3

7. List of partitions in consumer-groups is the same

kafka-consumer-groups --bootstrap-server broker:9092 --group my-consumer-group --describe
GROUP              TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                HOST            CLIENT-ID
my-consumer-group  my-topic        0          7               7               0               kafka-975c4e8e-ae7a-437c-988e-c740923e5834 /10.116.1.147   kafka

Actual behavior: data is read from only one partition.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[Gourds]

+1