kafka-connect-transform-common
kafka-connect-transform-common copied to clipboard
Allow TimestampNowField to output TTL for DynamoDB
This adds the following options to the TimestampNowField transformer:
- add.amount "how many of the chosen ChronoUnits to add on top of the timestamp of the system."
- add.chronounit "String representation of the ChronoUnit to add, eg: 'DAYS'"
- target.type "The desired timestamp representation: Unix, Date"
All of these options have sane defaults to ensure backward compatibility. The idea behind the first two is that DynamoDB expects a TTL value per item inserted. The idea behind the last option is that the current implementation outputs time since epoch in milliseconds. DynamoDB expects a TTL value in Unix time, which is the time since epoch in seconds.
The current ValidEnum and ConfigUtils::getEnum don't agree with each other on which field to use for the ChronoUnit enum. One uses ChronoUnit::name the other uses the enum value.
We could handle this on 2 ways.
- Move
ValidChronoUnitto connect-utils this would mean we use values like "DAYS". - Make
ValidEnumandConfigUtils::getEnumplay nice withChronoUnitthis would mean we use values like "Days" From my limited experience with adding my own enum for the target type, it seems option 1 would be more in line with existing things.