ERR__ALL_BROKERS_DOWN consumer never comes back up

[jmbeach]

Environment Information

• OS: Arch
• Node Version: 16.14.2
• NPM Version: 8.5.0
• C++ Toolchain: g++
• node-rdkafka version: 2.12.0

Steps to Reproduce

• Start your consumer
• Shut down your kafka instance
• Wait until you get all brokers down
• Start your kafka instance back
• You can't consume messages any more

node-rdkafka Configuration Settings

{
  brokers: [...],
  protocol: sasl,
  sasl: {
    mechanism: plain,
    username: ...,
    password: ...
  },
  group.id: ...,
  event_cb: true
  enable.auto.commit: true
}

Additional context

In my development and qa environment for the project I'm working on, I randomly receive ERR__ALL_BROKERS_DOWN after maybe a day or so of running. After this happens, the consumer never comes back up and no more messages are consumed.

Am I doing something wrong? Is there a way to recover from this event?

Using consumer-flow if that's relevant.

Also, it seems like librdkafka v1.9.0 has a change that fixes false positives with all brokers down. Will there be a release of node-rdkafka that upgrades to v1.9.0 any time soon?

[jmbeach]

Trying an approach where if that error is received, I start a timer to try to reconnect the consumer. Seems sloppy though

[jmbeach]

even that doesn't work. It still just never sees another message after reconnecting