Dynamically set the s3_key_format from the values retrieved from regex parser in S3 output plugin

[tusharkdev]

Is your feature request related to a problem? Please describe. We want to segregate the logs based on log records. Our log records has stream i.e stdout and stderr, we want to divide the log transaction based on stream. This usecase can be extend for dynamic segregation of logs based on string in log records.

Describe the solution you'd like We want to retrieve the values by using regex parser and use that values in the s3_key_format. It is similar to TAG usage. Expected solution is record accessor can access the value parsed by regex and use it in S3 output plugin s3_key_format - /splunkhec/raw/generic/fluentbit/$TAG/%H%M%S-$UUID (current solution) s3_key_format - /splunkhec/raw/generic/fluentbit/$VARIABLE/%H%M%S-$UUID (expected solution) In this case, the VARIABLE is stream

Describe alternatives you've considered Open log file and get the stream and based on the stream, send the data from S3 to Splunk.

[tusharkdev]

#feature-request #enhancement

[shadowshot-x]

I am also facing this issue. Can someone help?

[ashraf133]

Same here, i need to add variables in s3_key_format