Larger batch sizes impact the performance of Redpanda

[rkruze]

When running with larger batch size, like 15MB, Redpanda throughput drops.

Running on n1-standard-16 with 6 local SSD drives, and 6 nodes total. I'm running with both 131KB batch size and 15MB batch size with a message size of 50KB. I see the following:

25k msg/sec -

1000k msg/sec

Ran fio with 4k block sizes on one of the machines and saw the following:

WRITE: bw=1036MiB/s (1087MB/s), 259MiB/s-259MiB/s (271MB/s-272MB/s), io=911GiB (978GB), run=900001-900001msec

[mmaslankaprv]

@rkruze are you able to share your benchmark settings ? I am trying to reproduce this behavior with no luck.

[dotnwat]

This performance result is from a very long time ago. Likely we still have performance impact when batches sizes reach 15MB+ but we aren't explicitly trying to optimize for this case yet. We'll be tracking this elsewhere when it comes up agian.