logstash-input-s3-sns-sqs icon indicating copy to clipboard operation
logstash-input-s3-sns-sqs copied to clipboard

Cannot process empty .gz files

Open marlock9 opened this issue 3 years ago • 3 comments

We're using AWS ALB access logging to S3. Sometimes ALB uploads empty (0b) files with .gz extension and S3 notifies about them to SQS. As result, this input plugin tries to process this files as gzipped, but always fails with Incomplete message in read_file. We´ll throw skip_delete. and start processing after visibility_timeout reached. This loop only ends on reaching message retention period in SQS (which is 4 days by default). I'll address PR to fix this soon.

marlock9 avatar Feb 20 '22 12:02 marlock9

Thank you very much, I’ll test & release next week. As a quickfix you should configure a DLQ, to filter these messages fast.

see: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue.html

regards Christian

cherweg avatar Feb 20 '22 19:02 cherweg

Has this problem been fixed?

psdjc avatar May 23 '22 05:05 psdjc

As you can see, PR is still not merged. Now you can manually build plugin from my repo and install it.

marlock9 avatar May 23 '22 18:05 marlock9