fluent-plugin-bigquery icon indicating copy to clipboard operation
fluent-plugin-bigquery copied to clipboard

Cannot see error message for insert

Open dobesv opened this issue 4 years ago • 0 comments

Environments

  • fluentd version: 1.11.1
  • plugin version: 2.2.0

Configuration

<label @bigquery>
  <filter **>
    @type record_modifier
    <record>
      browserVersion "${record['browser'] && record['browser']['version']}"
      operationSystemVersion "${record['operatingSystem'] && record['operatingSystem']['version']}"
      browser "${record['browser'] && record['browser']['name']}"
      operatingSystem "${record['operatingSystem'] && record['operatingSystem']['name']}"
    </record>
  </filter>
  <match **>
    @id bigquery
    @type bigquery_insert
    @log_level info

    auth_method json_key
    json_key /etc/fluent/secrets/bigquery-service-account.json

    project "#{ENV['BIGQUERY_PROJECT']}"
    dataset "#{ENV['BIGQUERY_DATASET']}"
    table   events

    skip_invalid_rows true
    ignore_unknown_values true

    <inject>
      time_key timestamp
      time_type string
      time_format %FT%T%.%N:z
      utc true
    </inject>

    <buffer>
      @type file
      path /var/log/fluentd-buffers/bigquery.buffer
      total_limit_size 8GB
      overflow_action block
    </buffer>
    auto_create_table false
  </match>
</label>

Expected Behavior

If there's a problem inserting records into BigQuery, I can see in the logs some helpful information about what the problem is.

Actual Behavior

The logs indicate there is an error, but there's no detail provided. Instead the errors are replaced with #.

Log (if you have)

2020-06-30 20:04:42 +0000 [warn]: [bigquery] insert errors project_id="xxx" dataset="xx" table="events" insert_errors="[#<Google::Apis::BigqueryV2::InsertAllTableDataResponse::InsertError:0x00007f4b574a03e0 @errors=[#, #], @index=1>]"

dobesv avatar Jun 30 '20 20:06 dobesv