[Feature] "Simulate" Dead Letter for Kafka

[jakoss]

Is your feature request related to a problem? Please describe. As far as i understand the documentation - currently if i throw an exception in request handler it will create DLQ entry for rabbitmq, but for kafka it will just ack message. This basically mean that there are completely different behavior between providers and it's hard to change between providers while keeping handler logic the same.

Describe the solution you'd like Kafka does not have DLQ as a concept, but Brighter could just simulate that. So if anything in handler fails - the message and error details can be passed to another topic with specified name.

Is that something you see useful?

[lillo42]

I don't think Brighter should support it, because Kafka is a stream system not a queue, you should handle this in your code, let see what other thinks

[jakoss]

Yes, i know it's not a native concept for Kafka, but i've seen a lot of systems that still use special topic for DLQ. It's just helpful to have it for "error log".

If you don't want to add it to Brighter that's completely fine, we can simulate it ourselves using pipeline i think :)

[iancooper]

I'm okay with a DLQ for a stream, since it can experience transient errors. Having a DLQ for a stream is standard. Kafka Connect and Streams API apps both support it. We just need to configure a DLQ topic and send to that.