nakadi icon indicating copy to clipboard operation
nakadi copied to clipboard

Queue publishing of events for the same key in a batch

Open a1exsh opened this issue 1 year ago • 0 comments

When hash partitioning (and/or log compaction) is used this approach ensures that the order of events is preserved for each key in the face of intermittent publishing errors.

For now we simply mark any events that were not attempted for publishing as aborted. Later this can be improved to implement retry of failed events.

No change of behavior when no event keys are set (random or user-defined partitioning): the whole batch is submitted as a single chunk.

a1exsh avatar Jul 19 '22 13:07 a1exsh