pulsar icon indicating copy to clipboard operation
pulsar copied to clipboard

[PIP-214][broker]Add broker level metrics statistics and expose to prometheus

Open yyj8 opened this issue 3 years ago • 0 comments

For detailed improvement instructions, please refer to issues: https://github.com/apache/pulsar/issues/18056

Motivation

Currently, pulsar does not statistics broker level metrics, and all relevant metrics are 0 by default.

When the number of topic partitions reaches more than 100000 or even millions, and the topic level metrics are exposed (exposeTopicLevelMetricsInPrometheus=true), if you want to query the metrics of the broker dimension, you need to summarize all topics, and the performance becomes very poor, or even the results cannot be queried from the promethus. Common broker level metrics include:

pulsar_topics_count
pulsar_subscriptions_count
pulsar_producers_count
pulsar_consumers_count
pulsar_rate_in
pulsar_rate_out
pulsar_throughput_in
pulsar_throughput_out
pulsar_storage_size
pulsar_storage_logical_size
pulsar_storage_write_rate
pulsar_storage_read_rate
pulsar_msg_backlog

We need to statistics the metrics of the broker dimension and expose them to prometheus to improve the performance of the monitoring of the broker dimension. Modify the original metrics name as follows:

pulsar_broker_topics_count
pulsar_broker_subscriptions_count
pulsar_broker_producers_count
pulsar_broker_consumers_count
pulsar_broker_rate_in
pulsar_broker_rate_out
pulsar_broker_throughput_in
pulsar_broker_throughput_out
pulsar_broker_storage_size
pulsar_broker_storage_logical_size
pulsar_broker_storage_write_rate
pulsar_broker_storage_read_rate
pulsar_broker_msg_backlog
  • [ ] doc
  • [x] doc-required
  • [ ] doc-not-needed
  • [ ] doc-complete

yyj8 avatar Oct 19 '22 13:10 yyj8