loss of data

[tracyliuzw]

Environments

• fluentd version:1.5.2
• plugin version:3.0.0

Configuration

Expected Behavior

Actual Behavior

Log (if you have)

I often encounter data loss in the process of using. The error log is as follows: 2022-11-20 22:44:07 -0500 [debug]: #2 Created new chunk chunk_id="5edf2de1add7ed91680a7b7b796c5a84" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=nil, tag=nil, variables={:topic=>nil}, seq=0> 2022-11-20 22:44:07 -0500 [debug]: #5 Created new chunk chunk_id="5edf2de1ae64c54bfa55d3b729307635" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=nil, tag=nil, variables={:topic=>nil}, seq=0> 2022-11-20 22:44:07 -0500 [debug]: #0 Created new chunk chunk_id="5edf2de1b684d9f971632dd4ddb60b42" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=nil, tag=nil, variables={:topic=>nil}, seq=0> 2022-11-20 22:44:07 -0500 [debug]: #0 64 messages send. 2022-11-20 22:44:08 -0500 [warn]: #0 insert errors project_id="doomsday-last-survivors-309306" dataset="gamedata" table="event_gamelog_1120" insert_errors="[#<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x00007fa01df22d48 @errors=[#<Google::Apis::BigqueryV2::ErrorProto:0x00007fa01df22370 @debug_info="", @location="event_time", @message="Invalid datetime string \"20827388\"", @reason="invalid">], @index=264>, #<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x00007fa01df20480 @errors=[#<Google::Apis::BigqueryV2::ErrorProto:0x00007fa01df1fa58 @debug_info="", @location="event_time", @message="Invalid datetime string \"20827388\"", @reason="invalid">], @index=266>, #<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x00007fa01df1dca8 @errors=[#<Google::Apis::BigqueryV2::ErrorProto:0x00007fa01df1d2a8 @debug_info="", @location="event_time", @message="Invalid datetime string \"20827388\"", @reason="invalid">], @index=268>, #<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x00007fa01df1b4f8 @errors=[#<Google::Apis::BigqueryV2::ErrorProto:0x00007fa01df1aa58 @debug_info="", @location="event_time", @message="Invalid datetime string \"20827388\"", @reason="invalid">], @index=270>, #<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x00007fa01df18c80 @errors=[#<Google::Apis::BigqueryV2::ErrorProto:0x00007fa01df18258 @debug_info="", @location="event_time", @message="Invalid datetime string \"20827388\"", @reason="invalid">], @index=272>, #<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x00007fa01df164a8 @errors=[#<Google::Apis::BigqueryV2::ErrorProto:0x00007fa01df15ad0 @debug_info="", @location="event_time", @message="Invalid datetime string \"20827388\"", @reason="invalid">], @index=330>, #<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x00007fa01df07a98 @errors=[#<Google::Apis::BigqueryV2::ErrorProto:0x00007fa01df070c0 @debug_info="", @location="event_time", @message="Invalid datetime string \"20827388\"", @reason="invalid">], @index=332>, #<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x00007fa01df05270 @errors=[#<Google::Apis::BigqueryV2::ErrorProto:0x00007fa01df04848 @debug_info="", @location="event_time", @message="Invalid datetime string \"20827388\"", @reason="invalid">], @index=334>, #<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x00007fa01df02778 @errors=[#<Google::Apis::BigqueryV2::ErrorProto:0x00007fa01df01d78 @debug_info="", @location="event_time", @message="Invalid datetime string \"20827388\"", @reason="invalid">], @index=336>, #<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x00007fa01defd228 @errors=[#<Google::Apis::BigqueryV2::ErrorProto:0x00007fa01defbdd8 @debug_info="", @location="event_time", @message="Invalid datetime string \"20827388\"", @reason="invalid">], @index=338>]" 2022-11-20 22:44:08 -0500 [error]: #0 Hit limit for retries. dropping all chunks in the buffer queue. retry_times=0 records=414 error_class=Fluent::BigQuery::UnRetryableError error="failed to insert into bigquery(insert errors), and cannot retry" 2022-11-20 22:44:08 -0500 [error]: #0 suppressed same stacktrace 2022-11-20 22:44:08 -0500 [debug]: #1 Created new chunk chunk_id="5edf2de293c526e9e99cf5231cc871f7" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=nil, tag=nil, variables={:topic=>nil}, seq=0> 2022-11-20 22:44:08 -0500 [debug]: #3 119 messages send. 2022-11-20 22:44:08 -0500 [debug]: #1 170 messages send. 2022-11-20 22:44:08 -0500 [debug]: #3 Created new chunk chunk_id="5edf2de298c52e870444bb87caadf0ce" metadata=#<struct Fluent::Plugin::Buffer::Metadata timekey=nil, tag=nil, variables={:topic=>nil}, seq=0>