data-prepper icon indicating copy to clipboard operation
data-prepper copied to clipboard

Allow configurable file_pattern in s3 sink object_keys

Open graytaylor0 opened this issue 1 year ago • 1 comments

Is your feature request related to a problem? Please describe. The s3 sink defaults to https://github.com/opensearch-project/data-prepper/blob/c45ddb15f9fc8803655e4879d3d3494ede937b0d/data-prepper-plugins/s3-sink/src/main/java/org/opensearch/dataprepper/plugins/sink/s3/configuration/ObjectKeyOptions.java#L14 and is not configurable.

Describe the solution you'd like As a user of Data Prepper's s3 sink, I would like to configure the file_pattern to be different than the default value of events-%{yyyy-MM-dd'T'HH-mm-ss'Z'}

Additional context Add any other context or screenshots about the feature request here.

graytaylor0 avatar Feb 09 '24 17:02 graytaylor0

If we made this feature available, users may end up with pipelines that create S3 objects that can write over each other. This because we include some random values in the S3 objects to ensure they do not overwrite each other.

I agree that allowing users modify the file name may be useful. But, we should give them some restrictions. Some ideas:

  • Make this a file_name_prefix and let users configure the prefix for the file names.
  • Require the user provide certain files such as a ${uniqueId} so that files do not overwrite each other.

dlvenable avatar Feb 09 '24 17:02 dlvenable