Dynamic bucket_name via string interporation

[tommynsong]

We would like to see if bucket parameter can support interpolation. We would like to upload file to different bucket based on the value of a certain field in our log.

[danielmotaleite]

This is a must have feature, so we can organize logs in the s3 instead of a huge file with all log types. as a workaround, we must define manually in the logstash output one s3 output for each "variable" we need, with a bucket or prefix value hardcoded to each value

[danielmotaleite]

Sorry, actually the prefix did worked fine with logstash fields, i'm using this:

prefix => "output/%{[type]}/%{+YYYY}/%{+MM}/%{+dd}/" 

and it did created the correct folders in the S3, except on some minor events that didn't had the [type] (that i already fixed)

So it is just the bucket name that fails. So the workaround is of course, use a hardcoded bucket name and create dynamic folders using the prefix

[anitakrueger]

This also fails for the region. We use 3 different regions and based on where the log originally came from, it needs to go into a bucket with the same region. I've tried to add a metadata field and use that in the region configuration, but it plain fails like below:

s3 {
...
  region => "%{[@metadata][s3][region]}"
  bucket => "%{[@metadata][s3][bucket]}"
...
}

2019-07-17T10:47:38.223407468Z [2019-07-17T10:47:38,222][ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<URI::InvalidURIError: bad URI(is not URI?): https://s3.%{[@metadata][s3][region]}.amazonaws.com>, :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/st...

[marielcherkassky]

Maybe someone found a workaround ? It seems that the issue wasnt solved in version 7.7

[kellypug]

Could all the fields have the ability to be parameterized and use string interpolation? I'm in the same boat as above where we would like to parameterize our bucket names, but also, the KMS key id field.

[unwenliu]

Is there any progress on this issue i need use event's fields for bucket_name

[andytechdad]

Any update on this? We're having to run lots of logstash instances to copy logs to S3, its not really cost effective.

[ankimme]

This is a really important feature that should be implemented as soon as possible. Some vendors of S3 storage offer access policies only based on bucket and not prefix. Thus it is desired to have a possibility to dynamically set the bucket name.