kafka-connect-transform-common icon indicating copy to clipboard operation
kafka-connect-transform-common copied to clipboard

Allow TimestampNowField to output TTL for DynamoDB

Open acemoo opened this issue 2 years ago • 0 comments

This adds the following options to the TimestampNowField transformer:

  • add.amount "how many of the chosen ChronoUnits to add on top of the timestamp of the system."
  • add.chronounit "String representation of the ChronoUnit to add, eg: 'DAYS'"
  • target.type "The desired timestamp representation: Unix, Date"

All of these options have sane defaults to ensure backward compatibility. The idea behind the first two is that DynamoDB expects a TTL value per item inserted. The idea behind the last option is that the current implementation outputs time since epoch in milliseconds. DynamoDB expects a TTL value in Unix time, which is the time since epoch in seconds.

The current ValidEnum and ConfigUtils::getEnum don't agree with each other on which field to use for the ChronoUnit enum. One uses ChronoUnit::name the other uses the enum value. We could handle this on 2 ways.

  1. Move ValidChronoUnit to connect-utils this would mean we use values like "DAYS".
  2. Make ValidEnum and ConfigUtils::getEnum play nice with ChronoUnit this would mean we use values like "Days" From my limited experience with adding my own enum for the target type, it seems option 1 would be more in line with existing things.

acemoo avatar Jun 01 '23 07:06 acemoo