fluentd icon indicating copy to clipboard operation
fluentd copied to clipboard

file_chunk: add stricter checks for broken meta files

Open Watson1978 opened this issue 7 months ago • 0 comments

Which issue(s) this PR fixes: Fixes #

What this PR does / why we need it: This PR improves meta file corruption checking.

The meta file contains at least the following field values.

https://github.com/fluent/fluentd/blob/fa2eb58922e1c36f83bf1d5243b325a860f72864/lib/fluent/plugin/buffer/file_chunk.rb#L249-L254

It might be possible that the @size is 0. @unique_id, @created_at, and @modified_at are set when FileChunk is initialized, so they definitely have some values. I think these fields should be written in meta file.

So, this PR adds the id, c, and m fields check.

This PR reinforces https://github.com/fluent/fluentd/pull/1874.

Without this changes, it might causes following error when launch fluentd every time with broken meta file:

2025-06-06 12:11:26 +0900 [error]: unexpected error while checking flushed chunks. ignored. error_class=NoMethodError error="undefined method '<' for nil"
  2025-06-06 12:11:26 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin/output.rb:1479:in 'block in Fluent::Plugin::Output#enqueue_thread_run'
  2025-06-06 12:11:26 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin/buffer.rb:548:in 'block in Fluent::Plugin::Buffer#enqueue_all'
  2025-06-06 12:11:26 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin/buffer.rb:542:in 'Array#each'
  2025-06-06 12:11:26 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin/buffer.rb:542:in 'Fluent::Plugin::Buffer#enqueue_all'
  2025-06-06 12:11:26 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin/output.rb:1479:in 'Fluent::Plugin::Output#enqueue_thread_run'
  2025-06-06 12:11:26 +0900 [error]: /Users/watson/src/fluentd/lib/fluent/plugin_helper/thread.rb:78:in 'block in Fluent::PluginHelper::Thread#thread_create'

Docs Changes:

Release Note:

Watson1978 avatar Jun 09 '25 08:06 Watson1978