graylog2-server
graylog2-server copied to clipboard
Support Message size based output_batch_size
The output output_batch_size config parameter was always count based.
Limiting the batch size send to OS/ES based on count has the problem that the count might be too small to be efficient for small messages and too big (> 100MB) for huge messages.
So we end up having to customly tune this parameter for each scenario.
Supporting a byte size based limit would auto-tune the batch size instead.
Change our default from 500 messages to message batches of 50mb. A count based setting is still supported for backwards compatibility.
TODO
- Fix unit test
- Test thoroughly (maybe tweak the size calculation to reflect the actual serialized size of the message)
- Add changelog