fluent-plugin-cloudwatch-logs icon indicating copy to clipboard operation
fluent-plugin-cloudwatch-logs copied to clipboard

Revert backup chunk specification for suppressing useless chunk backup sets

Open bysnupy opened this issue 2 years ago • 0 comments

Description: Fluent::Unrecoverable Error makes the log set backup per a whole chunk unit, not only relevant log message. It even causes irrelevant logs lost due to not sending whole chunk sets for backup, in other words, unrecoverable error can block sending the whole chunk data set which has any bigger logs than 256kb. Ideally, we should skip only to send the certain bigger log message than 256kb, not all chunk data including multiple log messages.

In addition the chunk backup file would keep accumulating as useless things, because there is no solution to restore the chunk backup set binary files now. Refer https://github.com/fluent/fluentd/issues/3813 for potential risk of this behavior please.

bysnupy avatar Jul 11 '22 08:07 bysnupy