How do I handle failures on message processing coming from Kafka?

[MichalLechowski]

Hello,

as far as I can see there isn't any option to manually mark a message as processed correctly (Commit/CommitAsync?) and when a client (C# webjob) throws an error, the message is still commited and offset proceeds. Isn't that a road straight to losing messages on any failure on processing? Went through the docs and examples and I can't see anything about it.

My scenario is quite simple:

1. Get messages from Kafka through C# WebJob (Kafka input trigger)
2. Send messages to EventHub (EventHub output trigger)

But on any exception for whatever reason I need to save such message somewhere and handle it later. I know there is a retry policy that can repeat a certain message processing on failure (thrown exception) but that's not a good solution since I don't have any control over it e.g. I can set repeat 2 times and I can't get the information that I'm already on the last try and should save the loss of message somewhere if it doesn't success on the last try.

How should I do this? Am I missing something or should it be done 100% manually in the consumer app e.g. catch an exception, handle saving the failure somewhere, default auto commit message back to kafka?

Thanks for any suggestions.