data-prepper icon indicating copy to clipboard operation
data-prepper copied to clipboard

Dynamic key_path_prefix in S3 DLQ

Open graytaylor0 opened this issue 8 months ago • 1 comments

Is your feature request related to a problem? Please describe. As a user of the S3 DLQ, I would like to dynamically determine the prefix to send failed documents to with Data Prepper expressions, similar to how the S3 sink path_prefix can be dynamic

Describe the solution you'd like

sink:
  - opensearch:
       s3:
         dlq:
           key_path_prefix: "my-${/key}-prefix/"

Alternative Solution An alternative that would be more useful would be to create a pipeline DLQ that can send to the s3 sink, which supports dynamic path prefix

graytaylor0 avatar Jun 18 '24 17:06 graytaylor0