fluent-plugin-sql
fluent-plugin-sql copied to clipboard
Remove timezone on time format as it violated with BigQuery timeformat
Hi @repeatedly This PR aim to fix the error when streaming to BigQuery, related to this PR #45
<Google::Apis::BigqueryV2::ErrorProto:0x007fd4aa24d8b8 @debug_info=\“generic::invalid_argument: Could not parse ‘2017-03-13 07:21:09.177000+0000’ as a timestamp. Required format is YYYY-MM-DD HH:MM[:SS[.SSSSSS]]\“, @location=\“created_at\“, @message=\“Could not parse ‘2017-03-13 07:21:09.177000+0000’ as a timestamp. Required format is YYYY-MM-DD HH:MM[:SS[.SSSSSS]]\“, @reason=\“invalid\“>], @index=1>]”
Format should be configurable.
Thanks @repeatedly, I will fix it soon
By the way there is a BigQuery specific fluentd plugin that I have been using for a while now that seems to work very well.
Hi, any updates regarding this PR ?
I'm currently facing a timestamp issue which should be fixed by this change:
The comparison fails with the update_column fails on MySQL because of the timezone, as seen here:
ActiveRecord::StatementInvalid error="Mysql2::Error: Incorrect DATETIME value: '2021-09-30 06:43:14.000000+0000'
If I remove the timezone from the state file manually, it works fine (although it re-writes it incorrectly a second later)