[Meta, 8.4, 8.5] Logstash DLQ Improvements for 8.x

[jsvd]

Phase 1

• [x] [@kaisecheng] Don't overwrite the @metadata - #8721
• [x] Avoid waste of disk usage on each restart - #8022

Phase 2

• [x] Add retention mechanism - #6990 spec issue: #13915
  • [x] Add option to change behavior when DLQ is full PR #13923
    • [x] Add metrics. Described in issue #14010
    • [x] Add documentation (doc update already present in PR #13923)

Phase 3

• [x] Add DLQ (age) retention policy #14255
  • [x] Add metrics #14324
  • [x] Add documentation #14340
• [x] Allow deletion of DLQ contents after processing - #8904 meta issue #14173
  • [x] Add metrics #14336 https://github.com/logstash-plugins/logstash-input-dead_letter_queue/pull/45
  • [x] Add documentation #14341

Phase 4 Items under this list may or may not solve issues across plugins, or would require changes on other plugins. These items needs further discussion before completing.

• [x] DLQ re-triable errors in ES output https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/772
  • [x] [ES plugin] Add a new attribute to configure additional statuses to DLQ (currently only on 400 and 404) https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/697 , PR that solves it: https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1067.
  • [x] [ES plugin] Deprecate & rename confusingly named failure_type_logging_whitelist. PR that solves it: https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1068.
  • [x] [ES plugin] Consider sending events whose index has an interpolation failure to the DLQ, as suggested in DLQ when sprintf references a missing field #718 https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1084
• [x] [ES plugin] Consider introducing a new metric for events sent to the DLQ, as suggested in Introduce a new metric to track how many events go to the DLQ #778.

Phase 5

• [ ] Support DLQ in pipeline level exceptions (e.g. conditionals) initial draft work: https://github.com/elastic/logstash/pull/16218
• [ ] Implement DLQ support on more plugins - #7207
  • [ ] Prerequisite - [ES plugin] Avoid the nil setting and checking business. In the plugin the @dlq_writer is set to nil and when used it's checked for nil also discussed in #8064
• [ ] #15245

Discussion

• [ ] Implement DLQ as an Pipeline to Pipeline - #9502

[antoniocascais]

We are starting to use DLQs and we also have the problem of the log files not being deleted after being processes.

No idea how to help with those issues though, never worked on the logstash project before.

[eedugon]

I think https://github.com/elastic/logstash/issues/8795 should be added here too (cc: @jguay )

The current behavior when the DLQ is full is not clear and documented, and the ways to prune or clear data from the queue to make it working again is also unclear. That causes uncertainty and concerns.