tidwall
Upgrade kafka producer client
Using a webhook it's possible to send messages to Kafka. The documentation gives the following example:
SETHOOK warehouse kafka://10.0.20.78:9092/warehouse …
Almost all works as one would expect. One minor issue is that in the fast changing Kafka ecosystem the message timestamp mechanism has recently (a few years ago) been updated. Default LogAppendTime (broker creates timestamp) is replaced by default Creation Time (Kafka producer creates timestamp).
It seems that the Tile38 Kafka integration hook uses an old Kafka library. With Tile38 written in Go it is most probably the Go Kafka client. The old Go Kafka Client does not include timestamps, and with the current default Creation Time this results in records having timestamp value -1. This is unusable in downstream Kafka topics.
True, a work-around is to configure the Kafka topic to have log.message.timestamp.type=LogAppendTime, but it's rather inconvenient and a no longer the recommended way of working with Kafka event streaming.
Solution: would it be possible to upgrade the Go Kafka Client to a more recent version? This would align the Tile38 integration with the current Kafka ecosystem without creating legacy. Thanks!
I'm cool with updating the client. Do you know if this will cause any incompatibilities with legacy users?
Good question. I've had to dig a bit into the Kafka release notes to answer it. From what I can see the timestamp enhancement was introduced in the major release 1.0.0 in 2017. Check for log.message.timestamp.type here.
It seems to have been a change at the protocol level, which might not be forward compatible. Using log.message.format.version one can make the broker deal with older message versions during an upgrade, making it backward compatible. Not sure whether that works the other way around though.
We are already at version 2.5.x at this moment in time... time to upgrade! There'll be hardly anyone running such an old version of Kafka, as improvements are added continuously.
I just pushed an update that bumps the Kafka library to the latest version. https://github.com/Shopify/sarama v1.26.4
Thanks! As soon as there's a release/docker image available I'll test to see if it works.
With docker image tile38/tile38:edge still the same behavior.
timestampType 'create_time' is the default of my kafka broker. The -1 appears because the producer does not send a timestamp - as is expected using 'create_time'.
Is this issue perhaps related?
Makes sense. I just pushed an update to that assigns the minimum Kafka version to v0.10, based on instructions from the link you provided.