elastic
[discuss]Sensible defaults for age retention and clean consumed DLQ features
With version 8.4.0 of Logstash the age retention and clean consumed feature are available.
To make the transition as smooth as possible there were provided no default for age retention and disable clean consumed.
However, to provide the best user experience, the clean consumed feature (removal of already processed events from downstream DLQ input) should be enabled. This imposes us some thinking. The clean_consumed setting requires also that commit_offset is enabled. By default commit_offset is enabled, but if also clean_consumed is by default enabled and a user specifically disable commit_offset it reach a configuration error that stops the pipeline, because clean_consumed has a strong need on the commit_offset to be enabled. This would break an existing pipeline.
We have to thought as to shape this transition in a smoother way. One idea could be couple the 2 settings, so if commit_offset is enabled it implicitly enable also clean_consumed, if the first is disabled then also the second is without incurring in any subtle configuration error.
How about a setting for clean_consumed of auto? true if commit_offsets is true, and false if commit_offsets is false.
Introduce with the intention that, in a future release, this will become the default setting of clean_consumed. My guess is, that the number of users that want to keep consumed events and keep track of consumed events is very small
My guess is, that the number of users that want to keep consumed events and keep track of consumed events is very small
My feeling is the same, I like the auto proposal. So we could have true false and auto as possible values for clean_consumed setting, in a future minor release of DLQ input plugin, switch the default of clean_consumed from true to auto