data-prepper
data-prepper copied to clipboard
Have the option to write Event Metadata for failed documents to the DLQ
Is your feature request related to a problem? Please describe. As a user of Data Prepper's DLQ, I would like to also get the metadata of Events in the case that I was using those values in my pipeline for the failed DLQ data. This would allow me to replay DLQ data in the exact same way as the previous pipeline attempted.
Describe the solution you'd like
A configurable option at the dlq
level for include_event_metadata
, which would default to false.
dlq:
include_event_metadata: true
s3:
bucket: "dlq-bucket-ddb"
key_path_prefix: "osi-test-dlq-pipeline/logs/dlq/%{yyyy}/%{MM}/%{dd}"
region: "us-west-2"
sts_role_arn: "arn:aws:iam::870201406020:role/osis-to-domain-ingestion-role"
When set to true, the DLQ documents will contain an extra field for eventMetadata
, which contains the EventMetadata map.
Describe alternatives you've considered (Optional) Default all DLQ documents to contain the Event metadata
Additional context Add any other context or screenshots about the feature request here.