Unable to write after restarting kafka cluster

[xidiandb]

When I restarted the kafka cluster, the following problems occurred:

2020-01-17 07:29:35 +0000 [warn]: #0 failed to flush the buffer. retry_time=10 next_retry_seconds=2020-01-17 07:30:33 +0000 chunk="59c39b45987d2889fdf53f5e2d87f58c" error_class=Kafka::UnknownTopicOrPartition error="Kafka::UnknownTopicOrPartition"
  2020-01-17 07:29:35 +0000 [warn]: #0 suppressed same stacktrace
2020-01-17 07:29:45 +0000 [warn]: #0 Send exception occurred: Kafka::UnknownTopicOrPartition
2020-01-17 07:29:45 +0000 [warn]: #0 Exception Backtrace : /var/lib/gems/2.5.0/gems/ruby-kafka-0.6.8/lib/kafka/protocol.rb:130:in `handle_error'
/var/lib/gems/2.5.0/gems/ruby-kafka-0.6.8/lib/kafka/protocol/metadata_response.rb:144:in `partitions_for'
/var/lib/gems/2.5.0/gems/ruby-kafka-0.6.8/lib/kafka/cluster.rb:155:in `partitions_for'
/var/lib/gems/2.5.0/gems/fluent-plugin-kafka-0.7.9/lib/fluent/plugin/kafka_producer_ext.rb:190:in `assign_partitions!'
/var/lib/gems/2.5.0/gems/fluent-plugin-kafka-0.7.9/lib/fluent/plugin/kafka_producer_ext.rb:153:in `block in deliver_messages_with_retries'
/var/lib/gems/2.5.0/gems/fluent-plugin-kafka-0.7.9/lib/fluent/plugin/kafka_producer_ext.rb:148:in `loop'
/var/lib/gems/2.5.0/gems/fluent-plugin-kafka-0.7.9/lib/fluent/plugin/kafka_producer_ext.rb:148:in `deliver_messages_with_retries'
/var/lib/gems/2.5.0/gems/fluent-plugin-kafka-0.7.9/lib/fluent/plugin/kafka_producer_ext.rb:102:in `deliver_messages'
/var/lib/gems/2.5.0/gems/fluent-plugin-kafka-0.7.9/lib/fluent/plugin/out_kafka2.rb:220:in `write'
/var/lib/gems/2.5.0/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:1125:in `try_flush'
/var/lib/gems/2.5.0/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:1431:in `flush_thread_run'
/var/lib/gems/2.5.0/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'
/var/lib/gems/2.5.0/gems/fluentd-1.7.4/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-01-17 07:29:45 +0000 [warn]: #0 failed to flush the buffer. retry_time=10 next_retry_seconds=2020-01-17 07:30:46 +0000 chunk="59c39a5ff585997b0db7f41cc30228dd" error_class=Kafka::UnknownTopicOrPartition error="Kafka::UnknownTopicOrPartition"
  2020-01-17 07:29:45 +0000 [warn]: #0 suppressed same stacktrace
2020-01-17 07:29:45 +0000 [warn]: #0 Send exception occurred: Kafka::UnknownTopicOrPartition
2020-01-17 07:29:45 +0000 [warn]: #0 Exception Backtrace : /var/lib/gems/2.5.0/gems/ruby-kafka-0.6.8/lib/kafka/protocol.rb:130:in `handle_error'
/var/lib/gems/2.5.0/gems/ruby-kafka-0.6.8/lib/kafka/protocol/metadata_response.rb:144:in `partitions_for'
/var/lib/gems/2.5.0/gems/ruby-kafka-0.6.8/lib/kafka/cluster.rb:155:in `partitions_for'
/var/lib/gems/2.5.0/gems/fluent-plugin-kafka-0.7.9/lib/fluent/plugin/kafka_producer_ext.rb:190:in `assign_partitions!'
/var/lib/gems/2.5.0/gems/fluent-plugin-kafka-0.7.9/lib/fluent/plugin/kafka_producer_ext.rb:153:in `block in deliver_messages_with_retries'
/var/lib/gems/2.5.0/gems/fluent-plugin-kafka-0.7.9/lib/fluent/plugin/kafka_producer_ext.rb:148:in `loop'
/var/lib/gems/2.5.0/gems/fluent-plugin-kafka-0.7.9/lib/fluent/plugin/kafka_producer_ext.rb:148:in `deliver_messages_with_retries'
/var/lib/gems/2.5.0/gems/fluent-plugin-kafka-0.7.9/lib/fluent/plugin/kafka_producer_ext.rb:102:in `deliver_messages'
/var/lib/gems/2.5.0/gems/fluent-plugin-kafka-0.7.9/lib/fluent/plugin/out_kafka2.rb:220:in `write'
/var/lib/gems/2.5.0/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:1125:in `try_flush'
/var/lib/gems/2.5.0/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:1431:in `flush_thread_run'
/var/lib/gems/2.5.0/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'
/var/lib/gems/2.5.0/gems/fluentd-1.7.4/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-01-17 07:29:45 +0000 [warn]: #0 failed to flush the buffer. retry_time=11 next_retry_seconds=2020-01-17 07:30:50 +0000 chunk="59c39a6a7a6e508dc8d8df3078918470" error_class=Kafka::UnknownTopicOrPartition error="Kafka::UnknownTopicOrPartition"

[xidiandb]

It worked when I removed the buffer？But some logs are missing

[xidiandb]

my conf is：

    <match *.template>
      @type kafka2
      brokers kafka-0.kafka-svc:9092,kafka-1.kafka-svc:9092,kafka-2.kafka-svc:9092
      default_topic template
      max_send_retries -1
      required_acks 1
      compression_codec lz4
      <format>
        @type json
      </format>
      <buffer>
        @type file_single
        path /log/buffer/kafka/template
        total_limit_size 4G
        chunk_limit_size 15M
        retry_type periodic
        retry_wait 60
        flush_mode interval
        flush_thread_count 2
        flush_interval 10s
        retry_forever true
        overflow_action block
      </buffer>
    </match>

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days