Transactional.flow should emit only after a message is committed

[szymonm]

Short description

Currently Transactional.flow emits elements downstream only after they are acked by a Kafka broker, but not necessarily after they are committed. This is counter intuitive, because it may produce messages even if transaction that contains them is aborted or fails.

Speaking more practically, currently there is no way to verify that a record was successfully committed. This means that I cannot count the number of elements that were written to Kafka.

I also believe that this behavior is never described in the documentation and can be confusing to the users.

What are your thoughts around this @2m @ennru @seglo ?

Possible solutions

1. Change the behaviour of transactional flow to emit only committed elements.
2. Implement a new sink that will emit only after committing successfully.
3. Add a way to know which elements were committed.

[2m]

Good point.

Current tests that are counting the committed elements are doing that by starting a separate consumer flow, but that is quite ad-hoc and comes with its own bells and whistles.

I would go for the first solution. The current implementation which emits acked but not necessarily committed elements is the least desirable, so we should not keep those semantics.

@szymonm are you willing to work on this?

[szymonm]

I can try, but certainly not in the next two weeks... So if you want to pick it up earlier, feel free to grab it.

[2m]

Certainly. I'll let you know when I start work on this.

[szymonm]

It should be easier than I thought... looking into that