Metrics Discrepancies Between Produced and Consumed Messages in Kafka Confluent Platform

[MahBadran93]

Description: I am encountering discrepancies between the metrics displayed on the Confluent Platform website for my Kafka topics and the actual consumption and production counts observed in my environment. Specifically, after producing 2 messages to a topic, the "Consumption in the last hour" metric on Confluent's dashboard shows 7 messages consumed, which does not align with the expected count. Furthermore, in some instances, these metrics continue to increase even when no consumer application is actively running.

Could you please assist in understanding why these metrics might not accurately reflect the actual message counts being produced and consumed? This issue is crucial for monitoring and troubleshooting our Kafka infrastructure effectively.

How to produce: 1 - Produce Messages to Topic using client producer - I am using the confluent-kafka-python 2- Monitor metrics on confluent platform - Navigate to the Confluent Platform web interface. - Go to the specific Kafka topic where you produced the messages. - Observe the "Consumption in the last hour" metric or any other relevant metrics. 3-Compare Metrics with Produced Messages: - Note down the expected number of messages produced (e.g., 5 messages). - Compare this with the metrics displayed on the Confluent Platform dashboard. - Verify if the metrics accurately reflect the number of messages produced and consumed. 4- Check if the metric values change or increase over time, especially when no consumer applications are actively consuming messages.

Please have a look the screenshot below: it indicates that only 2 messages were produced to the topic, yet 7 messages were consumed from it. Upon checking my consumer client, it confirms that only two messages have been consumed. What could be causing this discrepancy? Is this issue originating from Confluent's side? Or There are issues with my configuration? or it is another thing?

I am not sure if I am doing something wrong or if this is a normal behaviour from confluent, and if it is normal behaviour what could be the extra messages being produced? do you know the best way to debug the issue with confluent-kafka-python?

More Information: 1 - confluent-kafka-python: 2.3.0 2 - librdkafka: ('2.3.0', 33751295) 3 - Consumer Client Configuration: - enable.auto.commit = False - auto.offset.reset = latest - session.timeout.ms=45000 - group.id -> only one consumer in the group - session.timeout.ms=45000 4 - we are using kafka cluster from confluent