fluent-plugin-bigquery icon indicating copy to clipboard operation
fluent-plugin-bigquery copied to clipboard

Can use strftime-formatting with fluentd 1.x.x?

Open mugioka opened this issue 2 years ago • 1 comments

Environments

  • fluentd version: 1.14.6
  • plugin version: 2.3.0

Configuration

<label @BIGQUERY_STDOUT>
  <match docker.**>
    @type record_modifier
    remove_keys container_id,container_name,source
    tag ${tag.gsub('-','_')}
  </match>

  <match docker.*.stdout>
    @type bigquery_insert
    @id stdout_bigquery_insert_${tag[1]}

    auth_method json_key
    json_key "#{ENV['BIGQUERY_JSON_KEY']}"
    project "#{ENV['BIGQUERY_PROJECT_NAME']}"
    dataset ${tag[1]}
    table   stdout_%Y%m
    auto_create_table true

    schema [
      {"name": "timestamp", "type": "TIMESTAMP"},
      {"name": "level", "type": "STRING"},
      {"name": "label", "type": "STRING"},
      {"name": "message", "type": "STRING"}
    ]

    <buffer tag,time>
      timekey 1d
      chunk_limit_records 300
      chunk_limit_size 768k
      queue_limit_length 5000
      flush_mode interval
      flush_interval 1s
      flush_thread_interval 0.05
      flush_thread_count 10
      flush_at_shutdown true
    </buffer>
  </match>
</label>

Expected Behavior

work fine.

Actual Behavior

output following error:

2022-06-10 13:40:35 +0900 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Parameter 'bigquery_insert: project=fuga:hoge=${tag[1]}/table=stdout_%Y%m/fetch_schema_table=/template_suffix=' doesn't have timestamp placeholders for timekey 86400"

Log (if you have)

mugioka avatar Jun 10 '22 08:06 mugioka

It appears to work when I look at https://github.com/fluent-plugins-nursery/fluent-plugin-bigquery#strftime-formatting.

mugioka avatar Jun 10 '22 08:06 mugioka

I'm sorry. fluentd-core doesn't support detecting the monthly placeholder. I guess the reason why the duration varies by month.

anyway, the README description is wrong. In current fluentd, monthly time placeholder is not supported.

joker1007 avatar Oct 05 '22 14:10 joker1007

I see. Thank you!

mugioka avatar Oct 05 '22 22:10 mugioka