data-prepper
data-prepper copied to clipboard
Allow configurable file_pattern in s3 sink object_keys
Is your feature request related to a problem? Please describe. The s3 sink defaults to https://github.com/opensearch-project/data-prepper/blob/c45ddb15f9fc8803655e4879d3d3494ede937b0d/data-prepper-plugins/s3-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/s3/configuration/ObjectKeyOptions.java#L14 and is not configurable.
Describe the solution you'd like
As a user of Data Prepper's s3 sink, I would like to configure the file_pattern to be different than the default value of events-%{yyyy-MM-dd'T'HH-mm-ss'Z'}
Additional context Add any other context or screenshots about the feature request here.
If we made this feature available, users may end up with pipelines that create S3 objects that can write over each other. This because we include some random values in the S3 objects to ensure they do not overwrite each other.
I agree that allowing users modify the file name may be useful. But, we should give them some restrictions. Some ideas:
- Make this a
file_name_prefix
and let users configure the prefix for the file names. - Require the user provide certain files such as a
${uniqueId}
so that files do not overwrite each other.