fluent-plugin-sql icon indicating copy to clipboard operation
fluent-plugin-sql copied to clipboard

Remove timezone on time format as it violated with BigQuery timeformat

Open phongtattuan opened this issue 8 years ago • 4 comments

Hi @repeatedly This PR aim to fix the error when streaming to BigQuery, related to this PR #45

<Google::Apis::BigqueryV2::ErrorProto:0x007fd4aa24d8b8 @debug_info=\“generic::invalid_argument: Could not parse ‘2017-03-13 07:21:09.177000+0000’ as a timestamp. Required format is YYYY-MM-DD HH:MM[:SS[.SSSSSS]]\“, @location=\“created_at\“, @message=\“Could not parse ‘2017-03-13 07:21:09.177000+0000’ as a timestamp. Required format is YYYY-MM-DD HH:MM[:SS[.SSSSSS]]\“, @reason=\“invalid\“>], @index=1>]”

phongtattuan avatar Apr 10 '17 03:04 phongtattuan

Format should be configurable.

repeatedly avatar Apr 11 '17 10:04 repeatedly

Thanks @repeatedly, I will fix it soon

phongtattuan avatar Apr 14 '17 04:04 phongtattuan

By the way there is a BigQuery specific fluentd plugin that I have been using for a while now that seems to work very well.

dobesv avatar Jun 22 '20 23:06 dobesv

Hi, any updates regarding this PR ?

I'm currently facing a timestamp issue which should be fixed by this change:

The comparison fails with the update_column fails on MySQL because of the timezone, as seen here:

ActiveRecord::StatementInvalid error="Mysql2::Error: Incorrect DATETIME value: '2021-09-30 06:43:14.000000+0000'

If I remove the timezone from the state file manually, it works fine (although it re-writes it incorrectly a second later)

patrixr avatar Sep 30 '21 07:09 patrixr