Roko

Just to compare, running the following on Kafka on the same hardware, with flushing after every message I get the following: ``` ./rdkafka_performance -P -b 10.0.0.15 -t sfo2 -s 1024...

https://github.com/vectorizedio/redpanda/issues/977 looks to also be related.

With 21.4.15 We get somewhat better results: ``` | elapsed | msgs | bytes | rtt | dr | dr_m/s | dr_MB/s | dr_err | tx_err | outq | offset...

Created a build with #1575 and #1496. Then ran on i3.large instances: Redpanda: ``` | elapsed | msgs | bytes | rtt | dr | dr_m/s | dr_MB/s | dr_err...

The command used is: `./rdkafka_performance -P -b -t -s 1024 -a all -c 10000000 -r 1000000 -u` what machine type was this running on?

I reran this just now on v21.7.6 and got the following: ``` | elapsed | msgs | bytes | rtt | dr | dr_m/s | dr_MB/s | dr_err | tx_err...

Ok, so running v28.8.2-beta1 release I set the following: ``` kafka_qdc_enable: true kafka_qdc_max_depth: 8 kafka_qdc_idle_depth: 8 kafka_qdc_max_latency_ms: 20 rpc_server_tcp_recv_buf=102400 ``` I get the following on Redpanda: ``` ./rdkafka_performance -P -b...

I believe @jcsp setup is on a single machine with redpanda pinned to a single core. I've been running on 3 machines. The current setup is with 3 i3en.large instances...

Again for reference here is Kafka on 3 i3en.large instances version 2.8.0 with `flush.messages=1` ``` | elapsed | msgs | bytes | rtt | dr | dr_m/s | dr_MB/s |...

Just to see the difference with log flushing on vs off here is Kafka 2.8.0 with no flushing: ``` | elapsed | msgs | bytes | rtt | dr |...