brooklin icon indicating copy to clipboard operation
brooklin copied to clipboard

Add support for preserving source message timestamp

Open sanjay24 opened this issue 4 years ago • 10 comments

This feature allows Brooklin to make use of Kafka message's source timestamp when sending the event to destination cluster. If the mirrored topic is configured with message.timestamp.type set to CreateTime, the intent is to have the timestamp supplied by the application. This feature will help achieving this intention.

sanjay24 avatar Feb 27 '20 13:02 sanjay24

@celiakung @ahmedahamid could you please review this PR?

sanjay24 avatar Feb 28 '20 01:02 sanjay24

@sanjay24 Will do.

ahmedahamid avatar Feb 28 '20 01:02 ahmedahamid

thanks @ahmedahamid @somandal

sanjay24 avatar Mar 03 '20 01:03 sanjay24

What is the expected timeline for to review this change @ahmedahamid, @somandal? It would be a very useful feature.

mateuszmrozewski avatar Mar 11 '20 22:03 mateuszmrozewski

Will look into it by end of next week. @mateuszmrozewski

ahmedahamid avatar Mar 12 '20 22:03 ahmedahamid

@celiakung could you please check & approve the changes? Thanks

sanjay24 avatar Apr 03 '20 09:04 sanjay24

There is still some more work required on this. I'll get back with my discovery and changes

sanjay24 avatar Apr 08 '20 11:04 sanjay24

@ahmedahamid @celiakung @somandal KafkaMirrorMakerConnectorTask was translating the records to pass the source event's timestamp only when its timestamp type was "LogAppendTime". Do you know why was it done? I've updated the PR to pass the correct timestamp when the feature is enabled.

sanjay24 avatar Apr 14 '20 01:04 sanjay24

@ahmedahamid could we get moving on the review? Also, when is the next Brooklin release planned?

sanjay24 avatar Apr 28 '20 06:04 sanjay24

@ahmedahamid Pinging you to get this reviewed and move ahead on this!

sanjay24 avatar Jul 27 '20 22:07 sanjay24