twmb
kgo: add examples of commiting mark offsets and consuming with dlq
Hi @arlanram , Thanks for this PR. As a user, I would like to clarify a few points:
- Does the retry() function allow configuring the number of retry attempts?
- Does using retry() block the queue and delay the processing of other pending events?
In the Baeldung article on Spring Retry Kafka Consumer, it suggests using RetryableTopic as a more optimal solution.
I think it would be worth considering this approach before finalizing the PR
hi @hale249
- yes, it depends on you, you can bind it to your config or external value, this is just for example usage
- it does not block or delay, because a new goroutine spawns when retry happens
RetryableTopic it's itself the same mechanism as DLQ.
Hi @arlanram, Actually, I'm also quite concerned about the possibility of losing messages when the server goes down while a goroutine is processing a message.
Do you have any additional options to help prevent data loss?
@hale249
- you should manage partition hooks on revoke and lost cases
- use graceful handling of records
- use external data storages to make sure you can restart a consumer on desired state and be able to prevent duplications
hope it can be useful!
Thankiu @arlanram I think it should be handled to minimize message loss during processing. With your logic, if the server is off, the message won't be processed. You should add the solution below for comprehensive handling.
- If there's an error, push it to a Kafka error topic, and then a consumer will listen and process it.
- If there's an error, push it to Redis/DB, and a cronjob will process it gradually
@hale249 please make sure that this is just for example usage of DLQ and MarkCommitOffsets. it's not intended to implement full solution. by the way see other examples in directory, hope it will be helpful.
Please add the one comment & squash commits (or at the least, squash commits) -- looks good otherwise, will merge once squashed.
