Turn off customMetrics, performanceCounters and metrics collected by default

[lidkowiak]

Is your feature request related to a problem? Please describe. By default agent collects lots of metrics/data:

customMetric Loaded Class Count customMetric % Of Max Heap Memory Used customMetric Heap Memory Used (MB) customMetric Suspected Deadlocked Threads customMetric Current Thread Count customMetric GC Total Time customMetric GC Total Count customMetric HeartbeatState performanceCounter Available Bytes performanceCounter IO Data Bytes/sec performanceCounter % Processor Time performanceCounter Private Bytes metric Loaded Class Count metric % Of Max Heap Memory Used metric Heap Memory Used (MB) metric Suspected Deadlocked Threads metric Current Thread Count metric GC Total Time metric GC Total Count metric HeartbeatState metric Available Bytes metric % Processor Time metric Private Bytes metric IO Data Bytes/sec

I prefer to use standard provided by Spring Boot/Micrometer. Is there a possibility to turn off collecting of the above telemetries?

Describe the solution you would like

By default no telemetries should be collected by default. Adding a config entry like collectStandardJVMMetric should cause collecting the data.

Describe alternatives you have considered

Additional context

[heyams]

{
  "instrumentation": {
        "jms": {
               "enabled": false
        }
}

have you tried this? this will disable all jms metrics, perf counters.

[heyams]

another option you can try is using metrics filtering. this allows you to filter out unwanted metrics.

[lidkowiak]

Well, I'm asking about JVM metrics plus I'm not using JMS atm so "jms enabled" will not solve it.

I've also found this https://github.com/microsoft/ApplicationInsights-Java/blob/9d1ec8cd6463673cfc5f6e1e8d458afe9a284bbb/agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/init/TelemetryClientInitializer.java#L91 It suggests that it's not possible to disable them as they are treated as non-filtered metrics/perf counters.

[heyams]

default metrics and perf counter collected by 3.x agent. you can use metrics filtering to filter both perf counter and custom metrics. i don't believe we use jvm to define any metrics we collect.

[heyams]

fyi:

some metrics specific to jvm are like the following: none of these matched what you had above.

| jvm_memory_used |     | jvm_buffer_memory_used |     | jvm_threads_states |     | jvm_memory_committed |     | jvm_gc_pause |     | jvm_memory_max |     | jvm_buffer_total_capacity |     | jvm_threads_live |     | jvm_classes_unloaded |     | jvm_classes_loaded |     | jvm_threads_peak |     | jvm_threads_daemon |     | jvm_buffer_count |     | jvm_gc_memory_promoted |     | jvm_gc_memory_allocated |     | jvm_gc_max_data_size |     | jvm_gc_live_data_size

[heyams]

here is an example how to use metric filter:

{
  "preview": {
    "processors": [
      {
        "type": "metric-filter",
        "exclude": {
          "matchType": "strict",
          "metricNames": [
            "a_test_metric",
            "another_test_metric"
          ]
        }
      }
    ]
  }
}

[lidkowiak]

I added following config:

"preview": {
    "processors": [
      {
        "type": "metric-filter",
        "exclude": {
          "matchType": "strict",
          "metricNames": [
            "Loaded Class Count",
            "% Of Max Heap Memory Used",
            "Heap Memory Used (MB)",
            "Suspected Deadlocked Threads",
            "Current Thread Count",
            "GC Total Time",
            "GC Total Count",
            "HeartbeatState",
            "\\Memory\\Available Bytes",
            "\\Process\\Private Bytes",
            "\\Process\\IO Data Bytes/sec",
            "\\Process\\% Processor Time",
            "\\Processor\\% Processor Time",
            "Available Bytes",
            "Private Bytes",
            "IO Data Bytes/sec",
            "% Processor Time"
          ]
        }
      }
    ]
}

and still I got in AI following custom metrics/perf counters

  | performanceCounter | Available Bytes | Memory |     | performanceCounter | Private Bytes | Process |     | performanceCounter | IO Data Bytes/sec | Process |     | performanceCounter | % Processor Time | Processor |     | performanceCounter | % Processor Time | Process |     | metric | Available Bytes |   |     | metric | Private Bytes |   |     | metric | IO Data Bytes/sec |   |     | metric | % Processor Time  |   |

I believe they match following code https://github.com/microsoft/ApplicationInsights-Java/blob/9d1ec8cd6463673cfc5f6e1e8d458afe9a284bbb/agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/init/TelemetryClientInitializer.java#L91

The fact that the agent collects some metrics/perf counters out-of-the box plus it is not documented at all is confusing. Additionally some telemetries can not be filtered.

I think metrics with prefix jvm_ comes from micrometer :) Magically when I put agent jar on the classpath they were collected

[heyams]

You can disable micrometer by doing the following:

{
  "instrumentation": {
    "micrometer": {
      "enabled": false
    }
  }
}

do you want to turn off metrics and perf counter completely? If yes, you can use sampling:

{
  "sampling": {
    "percentage": 0
  }
}

this will turn off all telemetries though.

alternatively, you can use metric interval:

{
  "preview": {
    "metricIntervalSeconds": 31536000   // captured once a year
  }
}

this includes perf counters and metrics. but it doesn't impact custom metrics via 2.x SDK trackMetrics.

Hope these help.

[heyams]

https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-in-process-agent#auto-collected-metrics https://github.com/microsoft/ApplicationInsights-Java/wiki/Default-Metrics-Captured-by-Java-3.0-Agent

we do documented auto-collected metrics. hope this will clear your confusion.

[lidkowiak]

Thanks, what I want to achieve is to turn off collecting of metrics/perf counters that are enabled by default. I partially achieved the goal with metric filtering but there are some leftovers. Obviously I still want to collect other telemetries, especially these that come from micrometer.

I think the scenario I want to achieve is pretty standard. I don't understand why there is something enabled by default that can not be completely turn off. Better approach would be to explicitly enable collecting of default metrics with some config.

I haven't found a link (https://github.com/microsoft/ApplicationInsights-Java/wiki/Default-Metrics-Captured-by-Java-3.0-Agent ) on docs.microsoft.com so I thought it's not documented at all :)

[heyams]

Thanks, a good point. We will make that link available in the public azure docs. I've created a work item to track it #12103176.

I think the scenario I want to achieve is pretty standard. I don't understand why there is something enabled by default that can not be completely turn off. Better approach would be to explicitly enable collecting of default metrics with some config.

i think that's a valid ask. I will circle back if we will extend this feature to let customers have the ability to turn off default collected custom metrics, perf counters and metrics.

[heyams]

@lidkowiak confirmed. we will add that feature to turn off custom metrics, metrics and perf counters collected by default. work item

[trask]

hi @lidkowiak @emmansun can you confirm your reason for wanting to turn of the default metrics below? is it for cost, or for other reasons? thx!

https://github.com/microsoft/ApplicationInsights-Java/blob/53ea6e7c0423968664a53ff128c384e96e608deb/agent/agent-tooling/src/main/java/com/microsoft/applicationinsights/agent/internal/telemetry/MetricFilter.java#L37-L45

[emmansun]

Hi @trask , in our use case, we use ApplicationInsights-Java for e2e trace, e.g. Request/Dependency/Trace log/..., for metric monitor, we have other solution. So we hope end user can have options to enable/disable those metrics collection, include default metrics. Yes, cost is also our concern. Thx!