fluent-bit
fluent-bit copied to clipboard
filter_rewrite_tag: fix in_emitter data loss when using ring buffers.
Summary
This PR seeks to avoid data loss when using the optional ring_buffer on emitters when using filter_rewrite_tag. To do so two new functions were added for the ring buffer API: peek and seek.
Description
During my work on #9919 I noticed that in_emitter had code to enable a ring buffer: https://github.com/fluent/fluent-bit/blob/cfc9ba95efc6769f820726c7031e07942f84db97/plugins/in_emitter/emitter.c#L221. This code has several problems:
- if an error occurs but not at the end of the loop it will be ignored.
- if a transient error occurs while ingesting the buffer it will be silently dropped.
- there is no way to activate the ring buffer at all.
This PR seeks to solve it by:
- adding the option
emitter_ring_buffer_sizetofilter_rewrite_tagto enable and set the size of the ring buffer for it'sin_emitter. - adding
flb_ring_buffer_peekandflb_ring_buffer_seekto theflb_ring_bufferAPI so buffers can be read then absorbed only once they have been successfully submitted.
I changed the use of flb_ring_buffer_read to flb_ring_buffer_peek and flb_ring_buffer_seek so that when the buffer is ingested it first calls peek to read the buffer then submit it and only seeks the buffer only if the buffer has been successfully ingested by flb_input_log_append, ie: it returns a non-error value.
The work adding the flb_ring_buffer APIs could and should be applied to the main ring buffer as well. If this is required now I can split out that part of the work into a separate PR and leave just the fixes for the emitter and filter_rewrite_tag here.
Enter [N/A] in the box, if an item is not applicable to your change.
Testing Before we can approve your change; please submit the following in a comment:
- [ ] Example configuration file for the change
- [ ] Debug log output from testing the change
- [ ] Attached Valgrind output that shows no leaks or memory corruption was found
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
- [ ] Run local packaging test showing all targets (including any new ones) build.
- [ ] Set
ok-package-testlabel to test for all targets (requires maintainer to do).
Documentation
- [ ] Documentation required for this feature
Backporting
- [ ] Backport to latest stable release.
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.