Support Pulsar producer and consumer metrics

[codelipenghui]

Background

The Apache Pulsar community will directly support OpenTelemetry metrics in the Pulsar client by adding the OpenTelemetry API dependency. However, it will require users to upgrade to upcoming releases (3.3.0). Users who want to stay on the old version also need a solution to expose Pulsar Client metrics via OpenTelemetry.

Motivation

This change will expose basic metrics of the Pulsar producers and consumers by default, and users can turn off the Metrics using JVM Args -Dotel.instrumentation.pulsar-clients-metrics.enabled=false. So that users can apply the Java Agent to their application to get Pulsar Client metrics from an old version of Pulsar Client (at least from 2.8.0 to 3.2.0).

The following metrics will be added to the Pulsar client instrumentation.

Producer Metrics

Metric Name	Attribute Keys	Unit	Metric Description	Metric Type
pulsar.client.producer.message.send.size	pulsar.topic pulsar.producer.name	bytes	Counts the size of sent messages	LONG_OBSERVABLE_GAUGE
pulsar.client.producer.message.send.count	pulsar.topic pulsar.producer.name pulsar.response.status	messages	Counts the number of sent messages	LONG_OBSERVABLE_GAUGE
pulsar.client.producer.message.send.duration	pulsar.topic pulsar.producer.name pulsar.quantile	ms	The duration of sent messages	DOUBLE_OBSERVABLE_GAUGE

Consumer Metrics

Metric Name	Attribute Keys	Unit	Metric Description	Metric Type
pulsar.client.consumer.message.received.size	pulsar.topic pulsar.subscription pulsar.consumer.name	bytes	Counts the size of received messages	LONG_OBSERVABLE_GAUGE
pulsar.client.consumer.message.received.count	pulsar.topic pulsar.subscription pulsar.consumer.name pulsar.response.status	messages	Counts the number of received messages	LONG_OBSERVABLE_GAUGE
pulsar.client.consumer.message.ack.count	pulsar.topic pulsar.subscription pulsar.consumer.name pulsar.response.status	acks	Counts the number of sent message acknowledgements	LONG_OBSERVABLE_GAUGE
pulsar.client.consumer.receiver.queue.usage	pulsar.topic pulsar.subscription pulsar.consumer.name	messages	Number of the messages in the receiver queue	LONG_OBSERVABLE_GAUGE

Implementation

This PR will introduced a new PulsarMetricsRegistry. The newly created producers and consumers will register to the metrics register and the metrics registry will collect the metrics from the producers and consumer by using Pulsar's public API producer.getStats() and consumer.getStats(), which exposed the basic metrics of the producer and consumer.

However, the exposed the metrics by the producer.getStats() and consumer.getStats() do not conformed to the OpenTelemetry metric types. For example, the getSendLatencyMillis50pct(); method will return the P50 publish latency directly which I can't build a real Summary or even Histogram. So I finally use Gauge for all the metrics exposed by Pulsar Client.

Integration tests

Tested the built java agent with the Pulsar perf client and the Otel receiver. The following is the output of the prometheus exporter.

Prometheus Exporter

# HELP jvm_class_count Number of classes currently loaded.
# TYPE jvm_class_count gauge
jvm_class_count{job="unknown_service:java"} 9662
# HELP jvm_class_loaded_total Number of classes loaded since JVM start.
# TYPE jvm_class_loaded_total counter
jvm_class_loaded_total{job="unknown_service:java"} 9663
# HELP jvm_class_unloaded_total Number of classes unloaded since JVM start.
# TYPE jvm_class_unloaded_total counter
jvm_class_unloaded_total{job="unknown_service:java"} 1
# HELP jvm_cpu_count Number of processors available to the Java virtual machine.
# TYPE jvm_cpu_count gauge
jvm_cpu_count{job="unknown_service:java"} 12
# HELP jvm_cpu_recent_utilization_ratio Recent CPU utilization for the process as reported by the JVM.
# TYPE jvm_cpu_recent_utilization_ratio gauge
jvm_cpu_recent_utilization_ratio{job="unknown_service:java"} 0.003794991544090168
# HELP jvm_cpu_time_seconds_total CPU time used by the process as reported by the JVM.
# TYPE jvm_cpu_time_seconds_total counter
jvm_cpu_time_seconds_total{job="unknown_service:java"} 19.13781
# HELP jvm_gc_duration_seconds Duration of JVM garbage collection actions.
# TYPE jvm_gc_duration_seconds histogram
jvm_gc_duration_seconds_bucket{job="unknown_service:java",jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation",le="0.01"} 8
jvm_gc_duration_seconds_bucket{job="unknown_service:java",jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation",le="0.1"} 8
jvm_gc_duration_seconds_bucket{job="unknown_service:java",jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation",le="1"} 8
jvm_gc_duration_seconds_bucket{job="unknown_service:java",jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation",le="10"} 8
jvm_gc_duration_seconds_bucket{job="unknown_service:java",jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation",le="+Inf"} 8
jvm_gc_duration_seconds_sum{job="unknown_service:java",jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation"} 0.024
jvm_gc_duration_seconds_count{job="unknown_service:java",jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation"} 8
# HELP jvm_memory_committed_bytes Measure of memory committed.
# TYPE jvm_memory_committed_bytes gauge
jvm_memory_committed_bytes{job="unknown_service:java",jvm_memory_pool_name="CodeHeap 'non-nmethods'",jvm_memory_type="non_heap"} 2.555904e+06
jvm_memory_committed_bytes{job="unknown_service:java",jvm_memory_pool_name="CodeHeap 'non-profiled nmethods'",jvm_memory_type="non_heap"} 4.325376e+06
jvm_memory_committed_bytes{job="unknown_service:java",jvm_memory_pool_name="CodeHeap 'profiled nmethods'",jvm_memory_type="non_heap"} 1.6121856e+07
jvm_memory_committed_bytes{job="unknown_service:java",jvm_memory_pool_name="Compressed Class Space",jvm_memory_type="non_heap"} 6.422528e+06
jvm_memory_committed_bytes{job="unknown_service:java",jvm_memory_pool_name="G1 Eden Space",jvm_memory_type="heap"} 1.17440512e+08
jvm_memory_committed_bytes{job="unknown_service:java",jvm_memory_pool_name="G1 Old Gen",jvm_memory_type="heap"} 1.6777216e+08
jvm_memory_committed_bytes{job="unknown_service:java",jvm_memory_pool_name="G1 Survivor Space",jvm_memory_type="heap"} 1.2582912e+07
jvm_memory_committed_bytes{job="unknown_service:java",jvm_memory_pool_name="Metaspace",jvm_memory_type="non_heap"} 5.111808e+07
# HELP jvm_memory_limit_bytes Measure of max obtainable memory.
# TYPE jvm_memory_limit_bytes gauge
jvm_memory_limit_bytes{job="unknown_service:java",jvm_memory_pool_name="CodeHeap 'non-nmethods'",jvm_memory_type="non_heap"} 5.849088e+06
jvm_memory_limit_bytes{job="unknown_service:java",jvm_memory_pool_name="CodeHeap 'non-profiled nmethods'",jvm_memory_type="non_heap"} 1.22912768e+08
jvm_memory_limit_bytes{job="unknown_service:java",jvm_memory_pool_name="CodeHeap 'profiled nmethods'",jvm_memory_type="non_heap"} 1.22896384e+08
jvm_memory_limit_bytes{job="unknown_service:java",jvm_memory_pool_name="Compressed Class Space",jvm_memory_type="non_heap"} 1.073741824e+09
jvm_memory_limit_bytes{job="unknown_service:java",jvm_memory_pool_name="G1 Old Gen",jvm_memory_type="heap"} 8.589934592e+09
# HELP jvm_memory_used_after_last_gc_bytes Measure of memory used, as measured after the most recent garbage collection event on this pool.
# TYPE jvm_memory_used_after_last_gc_bytes gauge
jvm_memory_used_after_last_gc_bytes{job="unknown_service:java",jvm_memory_pool_name="G1 Eden Space",jvm_memory_type="heap"} 0
jvm_memory_used_after_last_gc_bytes{job="unknown_service:java",jvm_memory_pool_name="G1 Old Gen",jvm_memory_type="heap"} 9.524736e+07
jvm_memory_used_after_last_gc_bytes{job="unknown_service:java",jvm_memory_pool_name="G1 Survivor Space",jvm_memory_type="heap"} 9.017768e+06
# HELP jvm_memory_used_bytes Measure of memory used.
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{job="unknown_service:java",jvm_memory_pool_name="CodeHeap 'non-nmethods'",jvm_memory_type="non_heap"} 1.454976e+06
jvm_memory_used_bytes{job="unknown_service:java",jvm_memory_pool_name="CodeHeap 'non-profiled nmethods'",jvm_memory_type="non_heap"} 4.078592e+06
jvm_memory_used_bytes{job="unknown_service:java",jvm_memory_pool_name="CodeHeap 'profiled nmethods'",jvm_memory_type="non_heap"} 1.3334016e+07
jvm_memory_used_bytes{job="unknown_service:java",jvm_memory_pool_name="Compressed Class Space",jvm_memory_type="non_heap"} 6.172912e+06
jvm_memory_used_bytes{job="unknown_service:java",jvm_memory_pool_name="G1 Eden Space",jvm_memory_type="heap"} 0
jvm_memory_used_bytes{job="unknown_service:java",jvm_memory_pool_name="G1 Old Gen",jvm_memory_type="heap"} 1.1527424e+08
jvm_memory_used_bytes{job="unknown_service:java",jvm_memory_pool_name="G1 Survivor Space",jvm_memory_type="heap"} 9.017768e+06
jvm_memory_used_bytes{job="unknown_service:java",jvm_memory_pool_name="Metaspace",jvm_memory_type="non_heap"} 5.0610424e+07
# HELP jvm_thread_count Number of executing platform threads.
# TYPE jvm_thread_count gauge
jvm_thread_count{job="unknown_service:java",jvm_thread_daemon="false",jvm_thread_state="runnable"} 1
jvm_thread_count{job="unknown_service:java",jvm_thread_daemon="false",jvm_thread_state="timed_waiting"} 2
jvm_thread_count{job="unknown_service:java",jvm_thread_daemon="false",jvm_thread_state="waiting"} 2
jvm_thread_count{job="unknown_service:java",jvm_thread_daemon="true",jvm_thread_state="runnable"} 4
jvm_thread_count{job="unknown_service:java",jvm_thread_daemon="true",jvm_thread_state="timed_waiting"} 11
jvm_thread_count{job="unknown_service:java",jvm_thread_daemon="true",jvm_thread_state="waiting"} 6
# HELP otlp_exporter_exported_total 
# TYPE otlp_exporter_exported_total counter
otlp_exporter_exported_total{job="unknown_service:java",success="false",type="log"} 126
otlp_exporter_exported_total{job="unknown_service:java",success="false",type="span"} 6008
# HELP otlp_exporter_seen_total 
# TYPE otlp_exporter_seen_total counter
otlp_exporter_seen_total{job="unknown_service:java",type="log"} 126
otlp_exporter_seen_total{job="unknown_service:java",type="span"} 6008
# HELP pulsar_client_consumer_acks_sent_count Counts the number of sent message acknowledgements
# TYPE pulsar_client_consumer_acks_sent_count gauge
pulsar_client_consumer_acks_sent_count{consumer_name="1048c",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-1"} 0
pulsar_client_consumer_acks_sent_count{consumer_name="1048c",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-1"} 248
pulsar_client_consumer_acks_sent_count{consumer_name="34bbd",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-5"} 0
pulsar_client_consumer_acks_sent_count{consumer_name="34bbd",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-5"} 248
pulsar_client_consumer_acks_sent_count{consumer_name="49393",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-6"} 0
pulsar_client_consumer_acks_sent_count{consumer_name="49393",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-6"} 247
pulsar_client_consumer_acks_sent_count{consumer_name="4ab67",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-4"} 0
pulsar_client_consumer_acks_sent_count{consumer_name="4ab67",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-4"} 248
pulsar_client_consumer_acks_sent_count{consumer_name="4b84e",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-9"} 0
pulsar_client_consumer_acks_sent_count{consumer_name="4b84e",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-9"} 248
pulsar_client_consumer_acks_sent_count{consumer_name="5509e",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-8"} 0
pulsar_client_consumer_acks_sent_count{consumer_name="5509e",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-8"} 247
pulsar_client_consumer_acks_sent_count{consumer_name="b771c",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-7"} 0
pulsar_client_consumer_acks_sent_count{consumer_name="b771c",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-7"} 248
pulsar_client_consumer_acks_sent_count{consumer_name="d4fd1",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-0"} 0
pulsar_client_consumer_acks_sent_count{consumer_name="d4fd1",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-0"} 247
pulsar_client_consumer_acks_sent_count{consumer_name="de87c",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-2"} 0
pulsar_client_consumer_acks_sent_count{consumer_name="de87c",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-2"} 247
pulsar_client_consumer_acks_sent_count{consumer_name="e3ff3",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-3"} 0
pulsar_client_consumer_acks_sent_count{consumer_name="e3ff3",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-3"} 247
# HELP pulsar_client_consumer_message_received_count_messages Counts the number of received messages
# TYPE pulsar_client_consumer_message_received_count_messages gauge
pulsar_client_consumer_message_received_count_messages{consumer_name="1048c",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-1"} 0
pulsar_client_consumer_message_received_count_messages{consumer_name="1048c",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-1"} 248
pulsar_client_consumer_message_received_count_messages{consumer_name="34bbd",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-5"} 0
pulsar_client_consumer_message_received_count_messages{consumer_name="34bbd",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-5"} 248
pulsar_client_consumer_message_received_count_messages{consumer_name="49393",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-6"} 0
pulsar_client_consumer_message_received_count_messages{consumer_name="49393",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-6"} 247
pulsar_client_consumer_message_received_count_messages{consumer_name="4ab67",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-4"} 0
pulsar_client_consumer_message_received_count_messages{consumer_name="4ab67",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-4"} 248
pulsar_client_consumer_message_received_count_messages{consumer_name="4b84e",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-9"} 0
pulsar_client_consumer_message_received_count_messages{consumer_name="4b84e",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-9"} 248
pulsar_client_consumer_message_received_count_messages{consumer_name="5509e",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-8"} 0
pulsar_client_consumer_message_received_count_messages{consumer_name="5509e",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-8"} 247
pulsar_client_consumer_message_received_count_messages{consumer_name="b771c",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-7"} 0
pulsar_client_consumer_message_received_count_messages{consumer_name="b771c",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-7"} 248
pulsar_client_consumer_message_received_count_messages{consumer_name="d4fd1",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-0"} 0
pulsar_client_consumer_message_received_count_messages{consumer_name="d4fd1",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-0"} 247
pulsar_client_consumer_message_received_count_messages{consumer_name="de87c",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-2"} 0
pulsar_client_consumer_message_received_count_messages{consumer_name="de87c",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-2"} 247
pulsar_client_consumer_message_received_count_messages{consumer_name="e3ff3",job="unknown_service:java",response_status="failure",subscription="sub",topic="persistent://public/default/test-3"} 0
pulsar_client_consumer_message_received_count_messages{consumer_name="e3ff3",job="unknown_service:java",response_status="success",subscription="sub",topic="persistent://public/default/test-3"} 247
# HELP pulsar_client_consumer_message_received_size_bytes Counts the size of received messages
# TYPE pulsar_client_consumer_message_received_size_bytes gauge
pulsar_client_consumer_message_received_size_bytes{consumer_name="1048c",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-1"} 253952
pulsar_client_consumer_message_received_size_bytes{consumer_name="34bbd",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-5"} 253952
pulsar_client_consumer_message_received_size_bytes{consumer_name="49393",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-6"} 252928
pulsar_client_consumer_message_received_size_bytes{consumer_name="4ab67",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-4"} 253952
pulsar_client_consumer_message_received_size_bytes{consumer_name="4b84e",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-9"} 253952
pulsar_client_consumer_message_received_size_bytes{consumer_name="5509e",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-8"} 252928
pulsar_client_consumer_message_received_size_bytes{consumer_name="b771c",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-7"} 253952
pulsar_client_consumer_message_received_size_bytes{consumer_name="d4fd1",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-0"} 252928
pulsar_client_consumer_message_received_size_bytes{consumer_name="de87c",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-2"} 252928
pulsar_client_consumer_message_received_size_bytes{consumer_name="e3ff3",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-3"} 252928
# HELP pulsar_client_consumer_receiver_queue_usage_messages Number of the messages in the receiver queue
# TYPE pulsar_client_consumer_receiver_queue_usage_messages gauge
pulsar_client_consumer_receiver_queue_usage_messages{consumer_name="1048c",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-1"} 0
pulsar_client_consumer_receiver_queue_usage_messages{consumer_name="34bbd",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-5"} 0
pulsar_client_consumer_receiver_queue_usage_messages{consumer_name="49393",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-6"} 0
pulsar_client_consumer_receiver_queue_usage_messages{consumer_name="4ab67",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-4"} 0
pulsar_client_consumer_receiver_queue_usage_messages{consumer_name="4b84e",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-9"} 0
pulsar_client_consumer_receiver_queue_usage_messages{consumer_name="5509e",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-8"} 0
pulsar_client_consumer_receiver_queue_usage_messages{consumer_name="b771c",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-7"} 0
pulsar_client_consumer_receiver_queue_usage_messages{consumer_name="d4fd1",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-0"} 0
pulsar_client_consumer_receiver_queue_usage_messages{consumer_name="de87c",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-2"} 0
pulsar_client_consumer_receiver_queue_usage_messages{consumer_name="e3ff3",job="unknown_service:java",subscription="sub",topic="persistent://public/default/test-3"} 0
# HELP pulsar_client_producer_message_sent_count_messages Counts the number of sent messages
# TYPE pulsar_client_producer_message_sent_count_messages gauge
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-72",response_status="failure",topic="test-9"} 0
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-72",response_status="success",topic="test-9"} 360
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-73",response_status="failure",topic="test-7"} 0
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-73",response_status="success",topic="test-7"} 360
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-74",response_status="failure",topic="test-8"} 0
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-74",response_status="success",topic="test-8"} 360
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-75",response_status="failure",topic="test-6"} 0
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-75",response_status="success",topic="test-6"} 360
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-76",response_status="failure",topic="test-5"} 0
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-76",response_status="success",topic="test-5"} 360
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-77",response_status="failure",topic="test-4"} 0
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-77",response_status="success",topic="test-4"} 360
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-78",response_status="failure",topic="test-3"} 0
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-78",response_status="success",topic="test-3"} 360
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-79",response_status="failure",topic="test-2"} 0
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-79",response_status="success",topic="test-2"} 360
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-80",response_status="failure",topic="test-1"} 0
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-80",response_status="success",topic="test-1"} 360
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-81",response_status="failure",topic="test-0"} 0
pulsar_client_producer_message_sent_count_messages{job="unknown_service:java",producer_name="standalone-0-81",response_status="success",topic="test-0"} 360
# HELP pulsar_client_producer_message_sent_duration_milliseconds The duration of sent messages
# TYPE pulsar_client_producer_message_sent_duration_milliseconds gauge
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-72",quantile="0.5",topic="test-9"} 2
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-72",quantile="0.75",topic="test-9"} 3
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-72",quantile="0.95",topic="test-9"} 4
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-72",quantile="0.99",topic="test-9"} 7
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-72",quantile="0.999",topic="test-9"} 7
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-72",quantile="max",topic="test-9"} 7
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-73",quantile="0.5",topic="test-7"} 3
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-73",quantile="0.75",topic="test-7"} 3
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-73",quantile="0.95",topic="test-7"} 5
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-73",quantile="0.99",topic="test-7"} 6
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-73",quantile="0.999",topic="test-7"} 6
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-73",quantile="max",topic="test-7"} 6
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-74",quantile="0.5",topic="test-8"} 3
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-74",quantile="0.75",topic="test-8"} 3
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-74",quantile="0.95",topic="test-8"} 6
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-74",quantile="0.99",topic="test-8"} 7
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-74",quantile="0.999",topic="test-8"} 7
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-74",quantile="max",topic="test-8"} 7
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-75",quantile="0.5",topic="test-6"} 2
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-75",quantile="0.75",topic="test-6"} 3
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-75",quantile="0.95",topic="test-6"} 4
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-75",quantile="0.99",topic="test-6"} 6
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-75",quantile="0.999",topic="test-6"} 6
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-75",quantile="max",topic="test-6"} 6
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-76",quantile="0.5",topic="test-5"} 3
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-76",quantile="0.75",topic="test-5"} 3
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-76",quantile="0.95",topic="test-5"} 4
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-76",quantile="0.99",topic="test-5"} 5
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-76",quantile="0.999",topic="test-5"} 5
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-76",quantile="max",topic="test-5"} 5
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-77",quantile="0.5",topic="test-4"} 3
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-77",quantile="0.75",topic="test-4"} 3
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-77",quantile="0.95",topic="test-4"} 4
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-77",quantile="0.99",topic="test-4"} 5
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-77",quantile="0.999",topic="test-4"} 5
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-77",quantile="max",topic="test-4"} 5
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-78",quantile="0.5",topic="test-3"} 3
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-78",quantile="0.75",topic="test-3"} 3
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-78",quantile="0.95",topic="test-3"} 4
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-78",quantile="0.99",topic="test-3"} 5
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-78",quantile="0.999",topic="test-3"} 5
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-78",quantile="max",topic="test-3"} 5
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-79",quantile="0.5",topic="test-2"} 3
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-79",quantile="0.75",topic="test-2"} 3
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-79",quantile="0.95",topic="test-2"} 4
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-79",quantile="0.99",topic="test-2"} 7
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-79",quantile="0.999",topic="test-2"} 7
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-79",quantile="max",topic="test-2"} 7
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-80",quantile="0.5",topic="test-1"} 2
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-80",quantile="0.75",topic="test-1"} 3
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-80",quantile="0.95",topic="test-1"} 4
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-80",quantile="0.99",topic="test-1"} 4
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-80",quantile="0.999",topic="test-1"} 4
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-80",quantile="max",topic="test-1"} 4
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-81",quantile="0.5",topic="test-0"} 2
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-81",quantile="0.75",topic="test-0"} 3
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-81",quantile="0.95",topic="test-0"} 4
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-81",quantile="0.99",topic="test-0"} 7
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-81",quantile="0.999",topic="test-0"} 7
pulsar_client_producer_message_sent_duration_milliseconds{job="unknown_service:java",producer_name="standalone-0-81",quantile="max",topic="test-0"} 7
# HELP pulsar_client_producer_message_sent_size_bytes Counts the size of sent messages
# TYPE pulsar_client_producer_message_sent_size_bytes gauge
pulsar_client_producer_message_sent_size_bytes{job="unknown_service:java",producer_name="standalone-0-72",topic="test-9"} 368640
pulsar_client_producer_message_sent_size_bytes{job="unknown_service:java",producer_name="standalone-0-73",topic="test-7"} 368640
pulsar_client_producer_message_sent_size_bytes{job="unknown_service:java",producer_name="standalone-0-74",topic="test-8"} 368640
pulsar_client_producer_message_sent_size_bytes{job="unknown_service:java",producer_name="standalone-0-75",topic="test-6"} 368640
pulsar_client_producer_message_sent_size_bytes{job="unknown_service:java",producer_name="standalone-0-76",topic="test-5"} 368640
pulsar_client_producer_message_sent_size_bytes{job="unknown_service:java",producer_name="standalone-0-77",topic="test-4"} 368640
pulsar_client_producer_message_sent_size_bytes{job="unknown_service:java",producer_name="standalone-0-78",topic="test-3"} 368640
pulsar_client_producer_message_sent_size_bytes{job="unknown_service:java",producer_name="standalone-0-79",topic="test-2"} 368640
pulsar_client_producer_message_sent_size_bytes{job="unknown_service:java",producer_name="standalone-0-80",topic="test-1"} 368640
pulsar_client_producer_message_sent_size_bytes{job="unknown_service:java",producer_name="standalone-0-81",topic="test-0"} 368640
# HELP queueSize_ratio The number of items queued
# TYPE queueSize_ratio gauge
queueSize_ratio{job="unknown_service:java",processorType="BatchLogRecordProcessor"} 0
queueSize_ratio{job="unknown_service:java",processorType="BatchSpanProcessor"} 92
# HELP system_disk_io_bytes_total System disk IO
# TYPE system_disk_io_bytes_total counter
system_disk_io_bytes_total{device="disk0",direction="read",job="unknown_service:java"} 1.176959913984e+12
system_disk_io_bytes_total{device="disk0",direction="write",job="unknown_service:java"} 1.24195878912e+12
system_disk_io_bytes_total{device="disk1",direction="read",job="unknown_service:java"} 1.8190336e+07
system_disk_io_bytes_total{device="disk1",direction="write",job="unknown_service:java"} 2.936832e+07
system_disk_io_bytes_total{device="disk2",direction="read",job="unknown_service:java"} 2.940928e+06
system_disk_io_bytes_total{device="disk2",direction="write",job="unknown_service:java"} 0
system_disk_io_bytes_total{device="disk3",direction="read",job="unknown_service:java"} 1.17693872128e+12
system_disk_io_bytes_total{device="disk3",direction="write",job="unknown_service:java"} 1.241929551872e+12
# HELP system_disk_operations_total System disk operations
# TYPE system_disk_operations_total counter
system_disk_operations_total{device="disk0",direction="read",job="unknown_service:java"} 7.7277709e+07
system_disk_operations_total{device="disk0",direction="write",job="unknown_service:java"} 5.5877178e+07
system_disk_operations_total{device="disk1",direction="read",job="unknown_service:java"} 3980
system_disk_operations_total{device="disk1",direction="write",job="unknown_service:java"} 2113
system_disk_operations_total{device="disk2",direction="read",job="unknown_service:java"} 238
system_disk_operations_total{device="disk2",direction="write",job="unknown_service:java"} 0
system_disk_operations_total{device="disk3",direction="read",job="unknown_service:java"} 7.7273478e+07
system_disk_operations_total{device="disk3",direction="write",job="unknown_service:java"} 5.5875065e+07
# HELP system_memory_usage_bytes System memory usage
# TYPE system_memory_usage_bytes gauge
system_memory_usage_bytes{job="unknown_service:java",state="free"} 7.244644352e+09
system_memory_usage_bytes{job="unknown_service:java",state="used"} 2.7115094016e+10
# HELP system_memory_utilization_ratio System memory utilization
# TYPE system_memory_utilization_ratio gauge
system_memory_utilization_ratio{job="unknown_service:java",state="free"} 0.2108469009399414
system_memory_utilization_ratio{job="unknown_service:java",state="used"} 0.7891530990600586
# HELP system_network_errors_total System network errors
# TYPE system_network_errors_total counter
system_network_errors_total{device="bridge100",direction="receive",job="unknown_service:java"} 0
system_network_errors_total{device="bridge100",direction="transmit",job="unknown_service:java"} 0
system_network_errors_total{device="en0",direction="receive",job="unknown_service:java"} 0
system_network_errors_total{device="en0",direction="transmit",job="unknown_service:java"} 0
# HELP system_network_io_bytes_total System network IO
# TYPE system_network_io_bytes_total counter
system_network_io_bytes_total{device="bridge100",direction="receive",job="unknown_service:java"} 10240
system_network_io_bytes_total{device="bridge100",direction="transmit",job="unknown_service:java"} 2.6605568e+07
system_network_io_bytes_total{device="en0",direction="receive",job="unknown_service:java"} 4.004538368e+09
system_network_io_bytes_total{device="en0",direction="transmit",job="unknown_service:java"} 1.798537216e+09
# HELP system_network_packets_total System network packets
# TYPE system_network_packets_total counter
system_network_packets_total{device="bridge100",direction="receive",job="unknown_service:java"} 47
system_network_packets_total{device="bridge100",direction="transmit",job="unknown_service:java"} 94752
system_network_packets_total{device="en0",direction="receive",job="unknown_service:java"} 1.07991469e+08
system_network_packets_total{device="en0",direction="transmit",job="unknown_service:java"} 8.8508519e+07

[linux-foundation-easycla[bot]]

The committers listed above are authorized under a signed CLA.

• :white_check_mark: login: codelipenghui / name: Penghui Li (e7edeea49b52764eca3c326dff0e6169203ea2ed, cb77997b2a9df24c174e4aaf69b930596d12f879, 3f5067589ec5a677b929f47d323274de280feb84, 78c998451a828e7e9456d9abd4d1ac9116bcc294)

[dao-jun]

@laurit @trask could you plz take a look?