recommendations for producer error handling

[pmidge]

Hi @alfred-landrum, now that our initial integration is done I'm doing some robustification work. I am considering stopping the external services that feed our producers if we detect an unrecoverable error talking with our brokers. The problem is I can't see how to do this given the current interface.

• producer.send does not appear to reject, though this may be a limitation of my understanding of Node addon code, which is basically none. I don't see anything obvious in ./src.
• trying to manually add error_cb via driver_options blows up in NaN.
• I hacked kafka-native locally to inject a callback for error_cb, replacing the default logging call.

The last gives me immediate feedback on the no-or-unhappy-broker case, which is what I want, but before further developing this I wanted guidance based on how you guys approached error handling when designing this thing. I'd prefer to extend in a direction in keeping with that.

If I'm missing something completely obvious, please let me know.

ETA: Best possible outcome would be an exception thrown during .send so this can be handled via standard Promise-based pattern.

[pmidge]

FYI - changes are in my fork if that factors into your feedback. First changeset was updated to include same functionality in consumer.

[alfred-landrum]

Hi @pmidge - There is no synchronous-like error reporting in the send side, and that was an intentional decision that worked ok with the original use case for the code. In that environment, we had no direct feedback to the upstream event creators, so even if kafka was down, there wasn't anyone that was synchronously listening for errors. Additionally, we wanted send to easily & with little overhead take multiple messages on a single call so that we could efficiently use the rdfkafka batch produce api, which also made us lean towards asynchronously reporting errors.

rdkafka does address this, so here's something you could try if you haven't already. There's a producer option called delivery_report_callback: https://github.com/alfred-landrum/node-kafka-native/blob/master/lib/producer.js#L23

With that in place, you should be able to get notified of send errors from rdkafka: https://github.com/alfred-landrum/node-kafka-native/blob/master/lib/producer.js#L41

I think it may take as long as 30 seconds by default for rdkafka to declare an error issue the callback; that's not currently configurable, but it could be.

Would you please try the delivery_report_callback option and see if that works for your use case?