bookkeeper icon indicating copy to clipboard operation
bookkeeper copied to clipboard

add flag to control whether basic metrics expose to prometheus or not

Open hangc0276 opened this issue 4 years ago • 4 comments

Motivation

When using prometheus-metrics-provider lib provided by BookKeeper, it will expose basic metrics such as cpu, jvm, memory metrics etc. However, other systems, eg. Apache Pulsar, may also expose those metrics, which will lead to those metrics register twice, and prometheus metrics provider thread will start failed.

Changes

  1. Add a flag to control whether basic metrics expose to prometheus or not.

hangc0276 avatar Mar 01 '21 12:03 hangc0276

@dlg99 I believe that the standard metrics cannot be renamed

            registerMetrics(new StandardExports());
            registerMetrics(new MemoryPoolsExports());
            registerMetrics(new GarbageCollectorExports());
            registerMetrics(new ThreadExports());

eolivelli avatar Mar 04 '21 08:03 eolivelli

@eolivelli can the names be changed there https://github.com/apache/bookkeeper/blob/5700641ea509be36a6395f56aae2f60a88fdcaab/bookkeeper-stats-providers/prometheus-metrics-provider/src/main/java/org/apache/bookkeeper/stats/prometheus/PrometheusMetricsProvider.java#L113

otherwise standard metrics can be extended / collect() overridden to provide overridden name https://github.com/prometheus/client_java/blob/7a6b9f1300c5445f477fb0452c5ec7b6f80464a1/simpleclient_hotspot/src/main/java/io/prometheus/client/hotspot/GarbageCollectorExports.java#L40

dlg99 avatar Mar 04 '21 21:03 dlg99

@eolivelli can the names be changed there

https://github.com/apache/bookkeeper/blob/5700641ea509be36a6395f56aae2f60a88fdcaab/bookkeeper-stats-providers/prometheus-metrics-provider/src/main/java/org/apache/bookkeeper/stats/prometheus/PrometheusMetricsProvider.java#L113

otherwise standard metrics can be extended / collect() overridden to provide overridden name https://github.com/prometheus/client_java/blob/7a6b9f1300c5445f477fb0452c5ec7b6f80464a1/simpleclient_hotspot/src/main/java/io/prometheus/client/hotspot/GarbageCollectorExports.java#L40

@dlg99 Thanks for you feedback. This method can override all the metrics' name when exposing to http port, but the metics name which registered into CollectorRegistry can't be override. Thus it will also lead to the same metric name registered twice and the prometheus metrics provider thread start failed. Thus, we should keep the flag to turn off standard metrics register and expose.

I also update the code and support adding metric prefix on metric name for standard metrics and pre-defined metrics. For user registered metric, and metric prefix won't be added.

@eolivelli @dlg99 PTAL thanks.

hangc0276 avatar Mar 28 '21 09:03 hangc0276

fix old workflow,please see #3455 for detail

StevenLuMT avatar Aug 24 '22 08:08 StevenLuMT