logstash-output-s3 icon indicating copy to clipboard operation
logstash-output-s3 copied to clipboard

upload_queue_size doesnt do anything

Open marielcherkassky opened this issue 4 years ago • 0 comments

Hey, I was trying to upload to s3 each message that I get from my input without doing any aggregation. Maybe I got it wrong but according to the docs upload_queue_size sounds fit for this issue - Number of items we can keep in the local queue before uploading them.

The output pipeline I configured :

output {
  s3 {
    region => "eu-east-1"
    bucket => "my_bucket"
    upload_queue_size => 1
    prefix => "files/"
    temporary_directory => "/etc/logstash/tmp_files_to_s3"
  }
  file {
     codec => rubydebug
     path => "/tmp/monitor.log"
  }
}

By default the rotation_strategy is configured to time or size thresholds. So I dont see how it behaves together with the upload_queue_size when it is configured.

marielcherkassky avatar Jun 03 '20 07:06 marielcherkassky