graylog2-server icon indicating copy to clipboard operation
graylog2-server copied to clipboard

Support Message size based output_batch_size

Open mpfz0r opened this issue 1 year ago • 0 comments

The output output_batch_size config parameter was always count based.

Limiting the batch size send to OS/ES based on count has the problem that the count might be too small to be efficient for small messages and too big (> 100MB) for huge messages.

So we end up having to customly tune this parameter for each scenario.

Supporting a byte size based limit would auto-tune the batch size instead.

Change our default from 500 messages to message batches of 50mb. A count based setting is still supported for backwards compatibility.

TODO

  • Fix unit test
  • Test thoroughly (maybe tweak the size calculation to reflect the actual serialized size of the message)
  • Add changelog

mpfz0r avatar May 31 '24 16:05 mpfz0r