data-prepper
data-prepper copied to clipboard
Allow using event fields in s3 sink object_key
Is your feature request related to a problem? Please describe. Currently it seems like all objects from the s3 sink are sent using the same prefix, with only date-time being configurable. This means in order to retrieve a subset of events, e.g. logs from a specific hostname, you need to query all events for the time period.
Describe the solution you'd like We would like to send events to different s3 object prefixes based on specific event fields, for example, hostname. This makes searching events in s3 simpler and cheaper as you can directly query the relevant subset of events.
Describe alternatives you've considered (Optional) We could potentially use separate sinks for each subset of logs but this is not really dynamic or scalable.
Additional context N/A