zookeeper icon indicating copy to clipboard operation
zookeeper copied to clipboard

ZOOKEEPER-4741: Modernize PrometheusMetricsProvider with Jetty and ne…

Open dukelion opened this issue 4 months ago • 3 comments

…w client

In production environments under heavy load, the existing PrometheusMetricsProvider can introduce high latency. This is largely due to its use of the outdated io.prometheus.client (simpleclient 0.x) library and its basic, built-in HTTPServer, which has limitations in server configuration and threading.

This commit modernizes the entire component to address these performance and maintainability issues by:

  1. Upgrading to the io.prometheus.metrics (client_java 1.x) library. This aligns the provider with the current standard for Prometheus instrumentation in Java and ensures future compatibility.

  2. Replacing the legacy server and threading model with an embedded Jetty server. The previous implementation used a custom thread pool for a metrics processing task queue, which is now obsolete in the new client library. This has been replaced with a robust Jetty server, which uses its own configurable thread pool to handle exporter servlet requests directly. This change improves stability and simplifies the threading model, resolving the latency issues under load.

These changes make the PrometheusMetricsProvider more stable, performant, maintainable, and easier to configure for production use.

dukelion avatar Aug 09 '25 15:08 dukelion

@kezhuw @anmolnar @eolivelli please have a look. I'm not sure why CI is failing.

UPD: just rebasing it fixed the test

dukelion avatar Aug 11 '25 15:08 dukelion

@dukelion Thanks! Please fix the compile error.

anmolnar avatar Oct 31 '25 16:10 anmolnar

@eolivelli @li4wang You might want to take a look at this patch before submission.

anmolnar avatar Dec 02 '25 00:12 anmolnar

Merged to master branch. Thanks @dukelion ! Please let me know your jira id, I'll assign the jira ticket to you.

anmolnar avatar Dec 12 '25 21:12 anmolnar