Adding example for kafka kraft metrics

[eskytthe]

Could also add this in a separate file if you like that approach better - or just as remarks in this file.

So this will add metrics for kafka.server:type=raft-metrics and kafka.server:type=broker-metadata-metrics - see https://kafka.apache.org/documentation/#kraft_monitoring. The currents patterns already cover kafka.controller:type=KafkaController.

Tested and running on kafka version 2.13-3.3.1

# HELP kafka_server_raft_metrics The average time in milliseconds to commit an entry in the raft log. kafka.server:name=null,type=raft-metrics,attribute=commit-latency-avg
# TYPE kafka_server_raft_metrics untyped
kafka_server_raft_metrics{name="commit-latency-avg",} NaN
kafka_server_raft_metrics{name="append-records-rate",} 0.0
kafka_server_raft_metrics{name="number-unknown-voter-connections",} 0.0
kafka_server_raft_metrics{name="election-latency-max",} NaN
kafka_server_raft_metrics{name="election-latency-avg",} NaN
kafka_server_raft_metrics{name="poll-idle-ratio-avg",} 0.34011684708415496
kafka_server_raft_metrics{name="current-vote",} -1.0
kafka_server_raft_metrics{name="commit-latency-max",} NaN
kafka_server_raft_metrics{name="high-watermark",} 1.8565754E7
kafka_server_raft_metrics{name="log-end-epoch",} 1424.0
kafka_server_raft_metrics{name="current-leader",} 2.0
kafka_server_raft_metrics{name="log-end-offset",} 1.8565755E7
kafka_server_raft_metrics{name="fetch-records-rate",} 2.013533586400396
kafka_server_raft_metrics{name="current-epoch",} 1424.0

# HELP kafka_server_broker_metadata The number of errors encountered by the BrokerMetadataPublisher while applying a new MetadataImage based on the latest MetadataDelta. kafka.server:name=null,type=broker-metadata-metrics,attribute=metadata-apply-error-count
# TYPE kafka_server_broker_metadata untyped
kafka_server_broker_metadata{name="metadata-apply-error-count",} 0.0
kafka_server_broker_metadata{name="last-applied-record-lag-ms",} 790.0
kafka_server_broker_metadata{name="last-applied-record-timestamp",} 1.682605212322E12
kafka_server_broker_metadata{name="metadata-load-error-count",} 0.0
kafka_server_broker_metadata{name="last-applied-record-offset",} 1.8390788E7

And we also get these:

# HELP kafka_server_raft_channel_metrics The total number of new connections established kafka.server:name=null,type=raft-channel-metrics,attribute=connection-creation-total
# TYPE kafka_server_raft_channel_metrics untyped
kafka_server_raft_channel_metrics{name="connection-creation-total",} 1.0
kafka_server_raft_channel_metrics{name="response-rate",} 2.005304353451064
kafka_server_raft_channel_metrics{name="select-rate",} 6.242318390581539
kafka_server_raft_channel_metrics{name="connection-close-total",} 0.0
kafka_server_raft_channel_metrics{name="network-io-rate",} 4.204670418526425
kafka_server_raft_channel_metrics{name="io-ratio",} 6.027092308687496E-4
kafka_server_raft_channel_metrics{name="io-time-ns-total",} 5.184752573E9
kafka_server_raft_channel_metrics{name="request-total",} 20053.0
kafka_server_raft_channel_metrics{name="successful-reauthentication-rate",} 0.0
kafka_server_raft_channel_metrics{name="io-wait-ratio",} 1.9730736790866161
kafka_server_raft_channel_metrics{name="outgoing-byte-rate",} 266.73461336138877
kafka_server_raft_channel_metrics{name="successful-authentication-rate",} 0.0
kafka_server_raft_channel_metrics{name="request-size-max",} 133.0
kafka_server_raft_channel_metrics{name="failed-authentication-rate",} 0.0
kafka_server_raft_channel_metrics{name="network-io-total",} 41682.0
kafka_server_raft_channel_metrics{name="incoming-byte-total",} 3208372.0
kafka_server_raft_channel_metrics{name="failed-reauthentication-rate",} 0.0
kafka_server_raft_channel_metrics{name="response-total",} 20052.0
kafka_server_raft_channel_metrics{name="incoming-byte-rate",} 318.8433921987192
kafka_server_raft_channel_metrics{name="io-wait-time-ns-total",} 2.0013717331819E13
kafka_server_raft_channel_metrics{name="connection-close-rate",} 0.0
kafka_server_raft_channel_metrics{name="request-size-avg",} 133.0
kafka_server_raft_channel_metrics{name="iotime-total",} 5.184752573E9
kafka_server_raft_channel_metrics{name="connection-creation-rate",} 0.0
kafka_server_raft_channel_metrics{name="successful-authentication-total",} 0.0
kafka_server_raft_channel_metrics{name="successful-authentication-no-reauth-total",} 0.0
kafka_server_raft_channel_metrics{name="connection-count",} 1.0
kafka_server_raft_channel_metrics{name="io-waittime-total",} 2.0013717331819E13
kafka_server_raft_channel_metrics{name="io-wait-time-ns-avg",} 3.160802694818653E8
kafka_server_raft_channel_metrics{name="reauthentication-latency-max",} NaN
kafka_server_raft_channel_metrics{name="failed-authentication-total",} 0.0
kafka_server_raft_channel_metrics{name="failed-reauthentication-total",} 0.0
kafka_server_raft_channel_metrics{name="io-time-ns-avg",} 96552.14507772021
kafka_server_raft_channel_metrics{name="request-rate",} 2.005523408732246
kafka_server_raft_channel_metrics{name="select-total",} 62068.0
kafka_server_raft_channel_metrics{name="successful-reauthentication-total",} 0.0
kafka_server_raft_channel_metrics{name="reauthentication-latency-avg",} NaN
kafka_server_raft_channel_metrics{name="outgoing-byte-total",} 2666969.0

[dhoard]

@eskytthe Thanks for the PR!

Since the most common pattern (at least initially) for using KRaft will be with standalone nodes (replacing ZK nodes), a KRaft-specific example would be great.

Do you want to add it to this PR or create a new one?

[SuperQ]

This needs a DCO sign-off. You can use git commit -s --amend to add it.

[eskytthe]

Just back from holiday will look at both things asap :-)

[dhoard]

@eskytthe any updates?