Allow AsyncProducer to delegate error handling

[ferrous26]

trafficstars

• Version of Ruby: 2.6.5
• Version of Kafka: 2.7
• Version of ruby-kafka: 1.3.0

Often times, the @braze-inc platform is producing messages at a rate such that if the brokers become unavailable, we will fill the buffers and need to either make new producers or drop messages. And once in a while we see timeouts during shutdown and don't want to just drop the data.

We'd like to make a backup system for these scenarios.

At the moment, we use the AsyncProducer, which will only report errors by passing the exception object to the @instrumenter, and may drop data if max_retries is set positively.

To address this, we were thinking of modifying AsyncProducer instrumentation to include the message that would have been dropped or adding an error_handler config option that takes a block to delegate handling of the error. We also would want to add some methods to clear out the MessageBuffer and PendingMessageQueue from from the underlying Producer and return whatever was in those buffers/queues.

We definitely wanted to get some feedback from the maintainers first about this line of thought.

[dasch]

I think you're better off building your own async producer – it's not a lot of code, as you can see from the ruby-kafka implementation, and would give you direct control.

[mensfeld]

I agree with this. On top of that you may want to look into rdkafka for messages production. It proved to be much more reliable with high volumes of messages produced.

[mattathompson]

Looking for something like this, I'm thinking of something for writing these messages to a persistent storage until brokers are available again.

[github-actions[bot]]

Issue has been marked as stale due to a lack of activity.