Can't push legacy buffer files automatic after fluentd restart

[breath-co2]

For some reason (for example:lost some fields of BigQuery table or network problem), the fluentd buffer have many files. After it restart, the legacy buffer files can't push into bigquery service automatic.

Environments

• fluentd version: 1.5.0
• plugin version:2.1.0

Configuration

  <system>
    workers 4
    root_dir "/data/fluentd-buffer/"
  </system>
  <match app.*>
    @type bigquery_insert
    @id bigquery
    auth_method json_key
    json_key xxxxxx
    project "xxxxxx"
    dataset "my_app"
    table "${tag[1]}"
    fetch_schema true
    request_timeout_sec 180
    <buffer tag>
      @type "file"
      flush_mode interval
      flush_interval 1
      queued_chunks_limit_size 1
      flush_at_shutdown true
      chunk_limit_records 10000
      chunk_limit_size 3m
      retry_type periodic
      retry_forever true
      retry_wait 1
      retry_timeout 24h
      retry_max_interval 3
      timekey_zone "+0700"
      flush_thread_count 32
      chunk_full_threshold 0.5
    </buffer>
  </match>

Expected Behavior

If buffer path have some buffers, It don't push the legacy buffer files when fluentd restart.

Actual Behavior

Push legacy buffer when fluentd restart.

Log (if you have)

ls -al worker*/bigquery/buffer/

worker0/bigquery/buffer/:
Total 114280
drwxr-xr-x 2 td-agent td-agent    8192 6月  10 16:58 .
drwxr-xr-x 3 td-agent td-agent      19 6月  10 16:17 ..
-rw-r--r-- 1 td-agent td-agent 2110968 6月  10 13:00 buffer.b58af10e888f24299d0aa41c931f2cab9.log
-rw-r--r-- 1 td-agent td-agent      89 6月  10 13:00 buffer.b58af10e888f24299d0aa41c931f2cab9.log.meta
-rw-r--r-- 1 td-agent td-agent 2106251 6月  10 13:00 buffer.b58af10e88b6f2265aff3329cfdf47c01.log
-rw-r--r-- 1 td-agent td-agent      89 6月  10 13:00 buffer.b58af10e88b6f2265aff3329cfdf47c01.log.meta
-rw-r--r-- 1 td-agent td-agent 2116685 6月  10 13:00 buffer.b58af10e88d7d9aed899ef623e09652ef.log
-rw-r--r-- 1 td-agent td-agent      89 6月  10 13:00 buffer.b58af10e88d7d9aed899ef623e09652ef.log.meta
...


worker1/bigquery/buffer/:
Total 271904
drwxr-xr-x 2 td-agent td-agent   20480 6月  10 16:58 .
drwxr-xr-x 3 td-agent td-agent      19 6月  10 16:17 ..
-rw-r--r-- 1 td-agent td-agent 1958231 6月  10 12:34 buffer.b58af0b2aa6cab7c88026be608263fc46.log
-rw-r--r-- 1 td-agent td-agent      89 6月  10 12:34 buffer.b58af0b2aa6cab7c88026be608263fc46.log.meta
-rw-r--r-- 1 td-agent td-agent 1957359 6月  10 12:34 buffer.b58af0b2aabb456128bf6d3fcf0eb632f.log
-rw-r--r-- 1 td-agent td-agent      89 6月  10 12:34 buffer.b58af0b2aabb456128bf6d3fcf0eb632f.log.meta
-rw-r--r-- 1 td-agent td-agent 1959182 6月  10 12:34 buffer.b58af0b2aadcf2f77260ad784d7115878.log
-rw-r--r-- 1 td-agent td-agent      89 6月  10 12:34 buffer.b58af0b2aadcf2f77260ad784d7115878.log.meta
.....

[joker1007]

I think that the responsibility of this probrem is fluentd itself. If buffer file already exists when fluentd is restarted, fluentd is supporsed to add bufferes to the processing queue. And fluentd renames a part of filename from b to q (that is marker for expressing the buffering status). But in this case, fluentd does not rename.