fluent-plugin-kafka icon indicating copy to clipboard operation
fluent-plugin-kafka copied to clipboard

kafka2 bugfix: Failed to assign partitions

Open leo198706 opened this issue 4 months ago • 0 comments

  1. fluent-plugin-kafka 0.18.1
  2. fluentd 1.14.6

error.log

According to the above error, I analyzed the reasons as follows:

When Kafka does not have a topic to write to and does not allow automatic creation, an error UnknownTopicOrPartition will be reported, but it will be caught by Kafka::Error https://github.com/fluent/fluent-plugin-kafka/blob/v0.18.1/lib/fluent/plugin/kafka_producer_ext.rb#L254

and finally thrown in rescue Exception => e. https://github.com/fluent/fluent-plugin-kafka/blob/master/lib/fluent/plugin/out_kafka2.rb#L411

This is not an ideal situation.

So I adjusted the code to handle the above situation and avoid repeated retries.

leo198706 avatar Sep 29 '24 03:09 leo198706