elastic
[Meta] DLQ removal of consumed events
Overview
This issue describes the ability of the DLQ to remove the consumed messages, so that there isn't required any manual intervention to keep only the messages not yet processed.
Implementation
The idea is to keep track of the last consumed event, leveraging the existing sincedb file to store the reference. Once a segment is completely consumed, it can be physically removed from the storage. The implementation involves both the input plugin and Logstash. The plugin is responsible to track the position, while on the Logstash side reside the responsibility to physically remove the segments once consumed. When the plugin starts, with the deletion setting enabled, it passes the position down to Logstash which keeps it in memory to understand when it’s needed to remove the physical file. On the plugin side, once a read from DLQ is completed, which generally means that the event has been inserted into the pipeline’s queue, the plugin has to acknowledge the DLQ readers through a callback. This is needed to coordinate the plugin and the DLQ reader so that the reader is able to know when a segment is fully read. The plugin shouldn’t be IO aggressive in updating the sinceDB file, so the writes should be batched. There could be identified some instants where the file could be flushed to disk:
- On close of the plugin, which is the actual policy.
- Once a segment is fully read. This requires coordination between the DLQ reader and the plugin, in particular the plugin needs to be notified of this condition, and so the best is that the plugin register for such an event.
- [optional] After a certain amount of events have been read. This could be done directly on the plugin side without any cooperation with the DLQ reader.
- [optional] On a timely basis. There should be a configurable interval, on plugin settings, after which the plugin automatically flushes the sincedb data, if there was some changes in the position after last check. It could be name sincedb_flush_interval and takes seconds as value.
Configuration
This behavior has to be explicitly enabled, on a per pipeline basis. By default it’s disabled, but when the logstash-input-dead_letter_queue plugin is configured with the setting clean_consumed => true, the DLQ reader stores the tail pointer in a persistent manner and is responsible for removal of completely consumed segments.
The DLQ input plugin has to verify the version of Logstash's core to check if it could use this functionality in case clean_consumed is enabled. If not, the plugin has to fail its registration phase.
Implementation Plan
- [x] [core] Drop tail segment when completely consumed, implementing adaptations to hook the acknowledgement of read and register a listener of a segment completion. PR #14188 bugfix for segment deletion PR #14274
- [x] [core] Use file locking on the reader if the clean consumed behavior is enabled. #14256
- [x] [plugin] Extract the sinceDB store and load into a separate entity. https://github.com/logstash-plugins/logstash-input-dead_letter_queue/pull/40
- [x] [plugin] Implement the logic to flush sincedb on segment completion. PR https://github.com/logstash-plugins/logstash-input-dead_letter_queue/pull/43
- [x] [core] Bugfix on writer to recognize correctly the queue full condition #14333
- [x] [metrics] expose metrics of delete segments (or the number of DLQ events cleaned) #14336 https://github.com/logstash-plugins/logstash-input-dead_letter_queue/pull/45
- [x] [doc] Update documentation (on LS core, where DLQ is described) to describe how this works and how to configure. #14341
