logstash
logstash copied to clipboard
reconsider max_inflight heuristic and warning message
Currently there's a MAX_INFLIGHT_WARN_THRESHOLD = 10_000 constant that is used to signal the user if it's likely that too many events will be in memory, and the formula is:
max_inflight = batch_size * pipeline_workers
Then we print:
@logger.warn("CAUTION: Recommended inflight events max exceeded! Logstash will run with up to #{max_inflight} events in memory in your current configuration. If your message sizes are large this may cause instability with the default heap size. Please consider setting a non-standard heap size, changing the batch size (currently #{batch_size}), or changing the number of pipeline workers (currently #{pipeline_workers})", default_logging_keys)
In master this formula may not be correct if you're using the memory queue, and is much less correct if using the persisted queue, where the high cap for events in memory is an unknown.
This warning must be reconsidered, either removed or made more accurate/robust.