Yet another example Kafka client configs

[LiamClarkeNZ]

@brian-brazil Two configurations I've deployed in production to capture essential (as recommended by the O'Reilly book's chapter on monitoring producers and consumers. Hopefully I got the metric names correct by Prometheus best practices, it's all quite new to me.

[brian-brazil]

The same comments apply here are in #305, KafkaMetricsReporter would be a better approach.

[LiamClarkeNZ]

@brian-brazil That's true where it's possible to do so. But a problem child of ours is Logstash, and it doesn't expose the ability to configure a metrics reporter for its consumers. In this instance, using the Java agent is the perfect solution.

[brian-brazil]

In that case I'd be looking to provide a config for logstash, not kafka client.

[LiamClarkeNZ]

In that case I'd be looking to provide a config for logstash, not kafka client.

It is a config for Logstash - the client config above is being used to export metrics for the Kafka client being used internally by Logstash.

[brian-brazil]

I'm not seeing any logstash mBeans there.

[LiamClarkeNZ]

I'm not seeing any logstash mBeans there.

Hi Brian, Logstash uses the kafka-client consumers to consume from Kafka and write to (in our case) ES, the Kafka consumers instantiated within Logstash register the metric MBeans I'm interested in to understand how Logstash is interacting with Kafka, so the configuration has to specify those metric mbeans.

[brian-brazil]

But this still isn't a logstash configuration. https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html#monitoring sounds like what you're looking for.

[LiamClarkeNZ]

Thanks @brian-brazil,

Unfortunately, the Logstash monitoring APIs do not expose specific Kafka consumer metrics that are highly relevant for diagnosing issues. Here's an example of the Kafka plugin metrics exposed by Logstash:

{
  "plugins": {
    "inputs": [
      {
        "id": "ssp_gauges",
        "events": {
          "out": 1156886854,
          "queue_push_duration_in_millis": 51341252
        },
        "name": "kafka"
      }
    ]
  }
}

However, for diagnosing problems with the Logstash consumers and how they interact with the Kafka cluster, metrics such as kafka.consumer<type=consumer-coordinator-metrics, client-id=(.+)><>sync-rate which are exposed via JMX, but not exposed by Logstash's monitoring API, are essential.

[brian-brazil]

That sounds like a feature request to Logstash then.

[brzyangg]

Hello, LiamClarkeNZ @LiamClarkeNZ I use your client configs, do not get metrics kafka version 2.2.0

[LiamClarkeNZ]

Kia ora @brzyangg, sorry for the very late reply - I hope you managed to resolve your issues :)

[LiamClarkeNZ]

@fstab @tomwilkie do you think using JMX exporter to monitor kafka client is worthwhile having as an example? If so, I can revisit it to ensure it works for Kafka 2.8 and 3.0, otherwise I'll close the PR :) It is an older use case now that Micrometer can export Kafka client metrics via Prometheus nicely, but I found value in it monitoring apps I couldn't easily change the code of.