fluent-plugin-kafka icon indicating copy to clipboard operation
fluent-plugin-kafka copied to clipboard

Published 20 hours ago • verified publisher icon fluent

Timestamp is always parsed as local time

Open wonnyz opened this issue 5 years ago • 1 comments

We use UTC timestamps when we write logs in Kafka, and date partitioning on S3. When I tried to parse a timestamp column like { "@t": "2020-11-11T07:58:58.7291622Z" }, generated filename was 20201110_22xx (GMT+9), not 20201111_07xx!

in-kafka-group creates TimeParser with just time_format, since there is no parameter like utc.

https://github.com/fluent/fluent-plugin-kafka/blob/fe1d71e59ddfd941c37714b5bf6bd1d2d9b15a50/lib/fluent/plugin/in_kafka_group.rb#L134-L140

TimeParser uses local time by default. (localtime = true) https://github.com/fluent/fluentd/blob/6757d70b7fdfcf5e4a6316b1ecd802dfeb0e326e/lib/fluent/time.rb#L208

Can we have utc parameter like parsers? When I set time_source as kafka, it worked properly. The timestamp was UTC.

wonnyz avatar Nov 11 '20 08:11 wonnyz

It seems that supporting utc may be good.

kenhys avatar Jul 05 '21 08:07 kenhys