kafka-monitor icon indicating copy to clipboard operation
kafka-monitor copied to clipboard

Published 20 hours ago • verified publisher icon linkedin

Parrallel Producer per partition

Open suyashtava opened this issue 2 years ago • 5 comments

Issue. #395

This PR aims to provide parallel Producer per partition, so that even if one partition is slow(hosted on slow Brtoker) other producers can easily produce data on other partitions.

suyashtava avatar Mar 22 '23 05:03 suyashtava

Previous Discussion: #375

suyashtava avatar Mar 22 '23 06:03 suyashtava

@suyashtava thanks for the PR, have your observed the memory consumption increase with this approach? Do you have some data to share?

mhratson avatar Mar 22 '23 21:03 mhratson

@suyashtava thanks for the PR, have you observed the memory consumption increase with this approach? Do you have some data to share?

@mhratson There is a small increase in memory. I don't have any comparison data ready at the moment since we changed this months ago in production.

GIve me a day I will reproduce the comparison data on the Weekend.

suyashtava avatar Mar 23 '23 15:03 suyashtava

@mhratson KMF is running on a Kafka Cluster with total of 12 Brokers, and each Broker is hosting 3 partitions of KMF.

With the existing opensource KMF: It is taking avg 73 MB of Memory, wherease with Parallel Producer(PR): It is taking avg 85 MB of memory.

CPU Usage almost remains untouched.

FYI: We are using java.lang.Memory.HeapMemoryUsage.usedBytes for memory calculation.

suyashtava avatar Mar 24 '23 19:03 suyashtava

Have you considered using using async send as alternative? That'd require changing metrics collection a bit rather than than constructing per-partition producer

mhratson avatar Mar 30 '23 15:03 mhratson