Need to understand the pattern for interal Queue ?

[abhit011]

I have been using kafka since 2017 and have been used librb c++ kafka lib since that time itself ...

Our produce rate on Kafka is like 3K -3.5k per second and each packet size if like max 260 bytes JSON .

Of-late there are some concerns where i did see that librbKafka internal queue keeps the messages for very long at times , where as when i check other Kafka producers process and its dump after 15 min gap, its like in some Kafka Producers processes it has reflected increase of internal queue , but not across all

e.g on kafka process ": { "name": "rdkafka#producer-1", "type": "producer", "ts":74918349890000, "time":1740642854, "replyq":0, "msg_cnt":12333, "msg_size":3012764, "msg_max":100000, "msg_size_max":1024000000, "simple_cnt":0, "metadata_cache_cnt":4, "brokers":{ "1.11.1:9093/0": { "name":"1.1.1.1:9093/0", "nodeid":0, "state":"UP", "stateage":2331906000, "outbuf_cnt":0, "outbuf_msg_cnt":0, "waitresp_cnt":700, "waitresp_msg_cnt":728, "tx":5781576, "txbytes":1944061754, "txerrs":0, "txretries":142266, "req_timeouts":705, "rx":5613850, "rxbytes":265996770, "rxerrs":0, "rxcorriderrs":0, "rxpartial":0, "zbuf_grow":0, "buf_grow":0, "wakeups":9927502, "int_latency": { "min":15000, "max":31000, "avg":15467, "sum":397928000, "cnt":25726 }, "rtt": { "min":15000, "max":104891000, "avg":35897086, "sum":946893344000, "cnt":26378 },

other kafka process dump more or less

{ "name": "rdkafka#producer-1", "type": "producer", "ts":74918369515000, "time":1740642874, "replyq":0, "msg_cnt":41, "msg_size":9970, "msg_max":100000, "msg_size_max":1024000000, "simple_cnt":0, "metadata_cache_cnt":4, "brokers":{ "1.1.1.1:9093/0": { "name":"1.1.1.1:9093/0", "nodeid":0, "state":"UP", "stateage":11337859000, "outbuf_cnt":0, "outbuf_msg_cnt":0, "waitresp_cnt":6, "waitresp_msg_cnt":6, "tx":510509, "txbytes":170707878, "txerrs":0, "txretries":3987, "req_timeouts":2, "rx":506211, "rxbytes":23909667, "rxerrs":0, "rxcorriderrs":0, "rxpartial":0, "zbuf_grow":0, "buf_grow":0, "wakeups":848129, "int_latency": { "min":15000, "max":31000, "avg":15617, "sum":325008000, "cnt":20810 }, "rtt": { "min":15000, "max":47172000, "avg":9539941, "sum":194538495000, "cnt":20392 }, "throttle": { "min":0, "max":0, "avg":0, "sum":0, "cnt":20391 }

whats the reason for the same

I do see that there are no rebalancing etc happen on kakfa brokers all were fine

its 7 node kafka cluster with 2 topic has 50 partition each

I need to understand the pattern whey its causing it ...