fluent-plugin-bigquery
fluent-plugin-bigquery copied to clipboard
Cannot see error message for insert
Environments
- fluentd version: 1.11.1
- plugin version: 2.2.0
Configuration
<label @bigquery>
<filter **>
@type record_modifier
<record>
browserVersion "${record['browser'] && record['browser']['version']}"
operationSystemVersion "${record['operatingSystem'] && record['operatingSystem']['version']}"
browser "${record['browser'] && record['browser']['name']}"
operatingSystem "${record['operatingSystem'] && record['operatingSystem']['name']}"
</record>
</filter>
<match **>
@id bigquery
@type bigquery_insert
@log_level info
auth_method json_key
json_key /etc/fluent/secrets/bigquery-service-account.json
project "#{ENV['BIGQUERY_PROJECT']}"
dataset "#{ENV['BIGQUERY_DATASET']}"
table events
skip_invalid_rows true
ignore_unknown_values true
<inject>
time_key timestamp
time_type string
time_format %FT%T%.%N:z
utc true
</inject>
<buffer>
@type file
path /var/log/fluentd-buffers/bigquery.buffer
total_limit_size 8GB
overflow_action block
</buffer>
auto_create_table false
</match>
</label>
Expected Behavior
If there's a problem inserting records into BigQuery, I can see in the logs some helpful information about what the problem is.
Actual Behavior
The logs indicate there is an error, but there's no detail provided. Instead the errors are replaced with #
.
Log (if you have)
2020-06-30 20:04:42 +0000 [warn]: [bigquery] insert errors project_id="xxx" dataset="xx" table="events" insert_errors="[#<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x00007f4b574a03e0 @errors=[#, #], @index=1>]"