data-prepper icon indicating copy to clipboard operation
data-prepper copied to clipboard

Allow using event fields in s3 sink object_key

Open cameronattard opened this issue 1 year ago • 9 comments

Is your feature request related to a problem? Please describe. Currently it seems like all objects from the s3 sink are sent using the same prefix, with only date-time being configurable. This means in order to retrieve a subset of events, e.g. logs from a specific hostname, you need to query all events for the time period.

Describe the solution you'd like We would like to send events to different s3 object prefixes based on specific event fields, for example, hostname. This makes searching events in s3 simpler and cheaper as you can directly query the relevant subset of events.

Describe alternatives you've considered (Optional) We could potentially use separate sinks for each subset of logs but this is not really dynamic or scalable.

Additional context N/A

cameronattard avatar Sep 07 '23 01:09 cameronattard