fluent-plugin-bigquery
fluent-plugin-bigquery copied to clipboard
Can use strftime-formatting with fluentd 1.x.x?
Environments
- fluentd version: 1.14.6
- plugin version: 2.3.0
Configuration
<label @BIGQUERY_STDOUT>
<match docker.**>
@type record_modifier
remove_keys container_id,container_name,source
tag ${tag.gsub('-','_')}
</match>
<match docker.*.stdout>
@type bigquery_insert
@id stdout_bigquery_insert_${tag[1]}
auth_method json_key
json_key "#{ENV['BIGQUERY_JSON_KEY']}"
project "#{ENV['BIGQUERY_PROJECT_NAME']}"
dataset ${tag[1]}
table stdout_%Y%m
auto_create_table true
schema [
{"name": "timestamp", "type": "TIMESTAMP"},
{"name": "level", "type": "STRING"},
{"name": "label", "type": "STRING"},
{"name": "message", "type": "STRING"}
]
<buffer tag,time>
timekey 1d
chunk_limit_records 300
chunk_limit_size 768k
queue_limit_length 5000
flush_mode interval
flush_interval 1s
flush_thread_interval 0.05
flush_thread_count 10
flush_at_shutdown true
</buffer>
</match>
</label>
Expected Behavior
work fine.
Actual Behavior
output following error:
2022-06-10 13:40:35 +0900 [error]: config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Parameter 'bigquery_insert: project=fuga:hoge=${tag[1]}/table=stdout_%Y%m/fetch_schema_table=/template_suffix=' doesn't have timestamp placeholders for timekey 86400"
Log (if you have)
It appears to work when I look at https://github.com/fluent-plugins-nursery/fluent-plugin-bigquery#strftime-formatting.
I'm sorry. fluentd-core doesn't support detecting the monthly placeholder. I guess the reason why the duration varies by month.
anyway, the README description is wrong. In current fluentd, monthly time placeholder is not supported.
I see. Thank you!