client_java icon indicating copy to clipboard operation
client_java copied to clipboard

Filter metrics by suffix

Open fstab opened this issue 3 years ago • 2 comments
trafficstars

Allow filtering metrics by suffix. For example, this can be used to filter out _created metrics.

There are different ways to configure this, depending on how you expose metrics:

Manual creation of the HTTPServer

    HTTPServer httpServer = new HTTPServer.Builder()
        .withRegistry(registry)
        .withSampleNameFilter(new SampleNameFilter.Builder()
            .nameMustNotEndWith("_created")
            .build())
        .build();

See TestHTTPServer.testCreatedMetricRemoved().

Exporter Servlet

    <servlet>
        <servlet-name>prometheus-exporter</servlet-name>
        <servlet-class>io.prometheus.client.servlet.jakarta.exporter.MetricsServlet</servlet-class>
        <init-param>
            <param-name>name-must-not-end-with</param-name>
            <param-value>_created</param-value>
        </init-param>
    </servlet>

jmx_exporter

We are working on a new configuration file format for jmx_exporter, see the new-config branch. The new config will have a section like this:

metricFilter:
  nameMustNotEndWith:
    - _created

fstab avatar May 20 '22 13:05 fstab

We are working on a new configuration file format for jmx_exporter, see the new-config branch. The new config will have a section like this:

This seems redundant, you can already configure the jmx exporter to drop samples by setting name to empty.

However you likely don't even need that - _created samples will generally only be there if there's mBeans exposing them and you've rules covering that with the JMX exporter, which wouldn't be typical at all..

brian-brazil avatar May 20 '22 14:05 brian-brazil

The main use case I had in mind for the metric filter was that some users want to filter out jvm_threads_deadlocked and jvm_threads_deadlocked_monitor because as the Javadoc says this might be an expensive operation under some circumstances. However, different users might want to filter different things, so I just exposed generic filtering by prefix/suffix/exact match.

One aspect that might be confusing: The name filters are applied after scraping the JMX bean. If you want to prevent the JMX bean from being scraped, there's the jmxBeanFilter (previously called blacklist / whitelist).

A complete example is test-config-new.yaml, but it's very early stages. I implemented a parser that's backwards-compatible with the current format. Next step is to write tests for different combinations of SSL config (both for the collector via jmxUrl and for the HTTPServer), but that's quite time consuming.

fstab avatar May 20 '22 15:05 fstab