logstash icon indicating copy to clipboard operation
logstash copied to clipboard

reconsider max_inflight heuristic and warning message

Open jsvd opened this issue 7 years ago • 0 comments

Currently there's a MAX_INFLIGHT_WARN_THRESHOLD = 10_000 constant that is used to signal the user if it's likely that too many events will be in memory, and the formula is:

max_inflight = batch_size * pipeline_workers

Then we print:

@logger.warn("CAUTION: Recommended inflight events max exceeded! Logstash will run with up to #{max_inflight} events in memory in your current configuration. If your message sizes are large this may cause instability with the default heap size. Please consider setting a non-standard heap size, changing the batch size (currently #{batch_size}), or changing the number of pipeline workers (currently #{pipeline_workers})", default_logging_keys)

In master this formula may not be correct if you're using the memory queue, and is much less correct if using the persisted queue, where the high cap for events in memory is an unknown.

This warning must be reconsidered, either removed or made more accurate/robust.

jsvd avatar Nov 09 '17 10:11 jsvd