redpanda-data
Transaction support
It would be nice to have support for transactions, so that you can see if a given record was part of a transaction or not, and what the state of the transaction is. Being able to see the transaction id would be a bonus (not to be confused with the transactional id of the producer!).
The information is available in the __transaction_state topic, which is created the first time a producer starts using transactions.
As it stands now, we have no way of telling whether or not a record will be consumed by a consumer running with isolation level ReadComitted.
The information could be displayed in a new column, or we could get an option to filter uncomitted records, or both.
Do you happen to know whether Kafka has an API for this as well? I assume otherwise we'd need to consume the topic on each instance and keep a lot information in RAM (or another persistence) available.
Similar to consumer group offsets the Kafka brokers offer an API to get this information straight from the Brokers.
I am not aware of any API that allows you to read that information. We use confluent-kafka-dotnet, which in turn uses librdkafka, and I can't find anything there.
This will be partially fixed with https://github.com/cloudhut/kowl/pull/154 . At least the part that shows whether a message was part of a transaction or not. I'm not sure whether we could track all transaction states as this would make Kowl stateful no? As far as I got that would require consuming the whole transaction state topic?
Seems like a DescribeTransactions API will soon be added: https://cwiki.apache.org/confluence/display/KAFKA/KIP-664%3A+Provide+tooling+to+detect+and+abort+hanging+transactions . Worth to wait for I think.