stream-reactor icon indicating copy to clipboard operation
stream-reactor copied to clipboard

Influxdb-sink connector: adding an extra field with sys_time()

Open afausti opened this issue 4 years ago • 1 comments

@stheppi in the influxdb-sink connector, we are currently using a KCQL query to extract the timestamp from a field:

INSERT INTO mytopic1 SELECT * FROM mytopic1 WITHTIMESTAMP producedTimeStamp

in our system produceTimeStamp corresponds to the time when the the data was produced.

It would be really useful if we could write sys_time() to an extra field to record the time when the point is written to InfluxDB. Then sys_time - procudedTimeStamp would help us to characterize the latency in our system.

Looking at the InfluxDB Sink connector documentation it seems that the only alternative is:

INSERT INTO mytopic1 SELECT * FROM mytopic1 WITHTIMESTAMP sys_time()

but then sys_time() is used as the InfluxDB timestamp and that is not what we want.

Is it possible to write a KCQL to add an extra field with sys_time()? if not, how much work is involved in changing the connector code to add this extra field?

afausti avatar Feb 22 '21 22:02 afausti

hey @stheppi do you have any comments on this issue? or any pointer to the connector code in case we want to add an extra field with the value of sys_time()?

afausti avatar Jun 10 '21 23:06 afausti