High memory usage even if no event is processed by the output plugin

[pracucci]

Today I've introduced the s3 output to our pipeline and I've noticed the plugin (and its dependencies) increases the logstash memory usage by ~100mb. The following screenshot shows the difference between with and without the s3 output. Please be aware that no event is processed by the s3 output during the tests done, so it's just having the plugin loaded that increases the memory usage.

I suspect it's mainly caused by aws-sdk, but it's the first time I dig into a logstash plugin and I may miss something. Does anyone have ever profiled the memory usage of the plugin (and its dependencies)? What's your take on this? Thanks!

---

• Version: 4.1.1
• Operating System: (Alpine) Linux
• Config File:

if [s3_access_log_tag] {
        s3 {
            bucket                              => "${ACCESS_LOGS_S3_BUCKET_NAME}"
            canned_acl                          => "private"
            encoding                            => "gzip"
            prefix                              => "%{type}-%{s3_access_log_tag}/year=%{+YYYY}/month=%{+MM}/day=%{+dd}"
            region                              => "${ACCESS_LOGS_S3_BUCKET_REGION}"
            time_file                           => 5
            rotation_strategy                   => "time"
            upload_workers_count                => 2
            storage_class                       => "STANDARD_IA"
            validate_credentials_on_root_bucket => false
            codec                               => json_lines
            restore                             => false
        }
}

Note: during tests no event had s3_access_log_tag field set, so no event was actually processed by s3 plugin.

[marioluan]

@pracucci

I personally don't see a problem with it if 1) you have enough RAM and 2) RAM doesn't increase exponentionally as you start to process events.

Why do you consider ~100MB of RAM increase "high"? Also, have you tried comparing the memory footprint for other plugins?

[pracucci]

you have enough RAM

It's a complex topic. Generally speaking, "enough RAM" used for system daemons means "less RAM" for applications. It's just a tradeoff.

Why do you consider ~100MB of RAM increase "high"?

Because in our setup, adding this plugin basically increased the logstash memory usage by +100%.

Also, have you tried comparing the memory footprint for other plugins?

I didn't compare it with other plugins doing the same stuff, but if - for example - we compare it with logstash-filter-translate the latter one has a non-significantly memory increase.