data-prepper icon indicating copy to clipboard operation
data-prepper copied to clipboard

Have the option to write Event Metadata for failed documents to the DLQ

Open graytaylor0 opened this issue 1 year ago • 0 comments

Is your feature request related to a problem? Please describe. As a user of Data Prepper's DLQ, I would like to also get the metadata of Events in the case that I was using those values in my pipeline for the failed DLQ data. This would allow me to replay DLQ data in the exact same way as the previous pipeline attempted.

Describe the solution you'd like A configurable option at the dlq level for include_event_metadata, which would default to false.

dlq:
    include_event_metadata: true
    s3:
          bucket: "dlq-bucket-ddb"
          key_path_prefix: "osi-test-dlq-pipeline/logs/dlq/%{yyyy}/%{MM}/%{dd}"
          region: "us-west-2"
          sts_role_arn: "arn:aws:iam::870201406020:role/osis-to-domain-ingestion-role"

When set to true, the DLQ documents will contain an extra field for eventMetadata, which contains the EventMetadata map.

Describe alternatives you've considered (Optional) Default all DLQ documents to contain the Event metadata

Additional context Add any other context or screenshots about the feature request here.

graytaylor0 avatar Feb 19 '24 20:02 graytaylor0