bad performance when executing kafka-producer-perf-test.sh

[lawrence-zhang]

Hi I got bad performance with kafka-proxy. currently we have 6 kafka brokers on our EKS and its has plaintext sasl. 3 kafka-proxy with SSL. the configuration as below - server - --debug-enable - --log-level=debug - --bootstrap-server-mapping=osskafka-kafka-0.osskafka-kafka-headless.infra.svc.cluster.local:9091,0.0.0.0:12687,kafka-proxy.hk.xxxx.dev.net:12687 - --bootstrap-server-mapping=osskafka-kafka-1.osskafka-kafka-headless.infra.svc.cluster.local:9091,0.0.0.0:12688,kafka-proxy.hk.xxxx.dev.net:12688 - --bootstrap-server-mapping=osskafka-kafka-2.osskafka-kafka-headless.infra.svc.cluster.local:9091,0.0.0.0:12689,kafka-proxy.hk.xxxx.dev.net:12689 - --bootstrap-server-mapping=osskafka-kafka-3.osskafka-kafka-headless.infra.svc.cluster.local:9091,0.0.0.0:12690,kafka-proxy.hk.xxxx.dev.net:12690 - --bootstrap-server-mapping=osskafka-kafka-4.osskafka-kafka-headless.infra.svc.cluster.local:9091,0.0.0.0:12691,kafka-proxy.hk.xxxx.dev.net:12691 - --bootstrap-server-mapping=osskafka-kafka-5.osskafka-kafka-headless.infra.svc.cluster.local:9091,0.0.0.0:12692,kafka-proxy.hk.xxxx.dev.net:12692 - --proxy-request-buffer-size=327680000 - --proxy-response-buffer-size=327680000 - --proxy-listener-read-buffer-size=327680000 - --proxy-listener-write-buffer-size=1310720000 - --kafka-connection-read-buffer-size=1310720000 - --kafka-connection-write-buffer-size=327680000 - --dynamic-listeners-disable - --proxy-listener-tls-enable - --proxy-listener-ca-chain-cert-file=/etc/kafka-proxy/secrets/ca.crt - --proxy-listener-cert-file=/etc/kafka-proxy/secrets/server.crt - --proxy-listener-key-file=/etc/kafka-proxy/secrets/private.key - --proxy-listener-key-password=XXXX - --kafka-max-open-requests=50000 - --producer-acks-0-disabled - --kafka-keep-alive=30m

we have expose endpoint with kafka-proxy.hk.xxxx.dev.net:12692 now we can use kafka-topics, kafka-console-producer and kafka-console-consumer normally. but when we executing performance testing, it is bad performance from pod inside.

org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120001 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120001 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120001 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120001 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120001 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120001 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120001 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-14:120001 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation org.apache.kafka.common.errors.TimeoutException: Expiring 8 record(s) for topic_performance_test-17:120002 ms has passed since batch creation 2128 records sent, 17.622459 records/sec (0.03 MB/sec), 161.69 ms avg latency, 463.00 ms max latency, 163 ms 50th, 195 ms 95th, 205 ms 99th, 208 ms 99.9th.

[cobolbaby]

Where is the bottleneck?

[lawrence-zhang]

if testing using kafka broker endpoints, performance is good. I don't know how to adjust configure to enhance performance

[lawrence-zhang]

I can't figure out the bottleneck.