Multiline Parser adds support for end_state and start flushing

[edmondsiu0]

Is your feature request related to a problem? Please describe. Yes. We use Fluent-bit for batch processing of logs. Some of these logs are pretty-printed spanning across multiple lines. We evaluated the use of multiline parser, but noticed the parser relies on flushing interval to get the in-memory item (generally speaking the last item in the log file) out of the multiline parser through the pipeline.

On the tail input plugin, we have set exit_on_eof which instructs fluent-bit to exit when it is done parsing the file. This exit signal causes fluent-bit to terminate, which leads to the still-in-memory multiline log to be abandoned unprocessed and lost.

Describe the solution you'd like For multiline parser rule, I would like to add support for end_state, so that I can signal the multiline parser that all lines have been captured and flush can immediately start, without having to wait for the flush interval.

Describe alternatives you've considered There is no workaround that I know of right now. Logs parsed by mutiline parser does not seem to subject to grace termination period, which always led to log lost.

Additional context Without this I am currently unable to use multiline parser as batch processor, as it always led to data loss.