Metric name "_info" suffix gets trimmed

[iagotomas]

trafficstars

Hi, trying to implement a ruleset for some service, running jmx_exporter 1.0.1 as standalone I'm facing a weird issue which I'm not sure I understand. I have the following rule for which the "_info" suffix in the name gets dropped. If I employ another suffix or add additional characters to the suffix "_info" it remains.

rules:
  #metrics:name=jvmInfo.runtime.{runtime}.vendor.{vendor}.version.{version},type=gauges
  - pattern: 'metrics<name=jvmInfo\.runtime\.(.+)\.vendor\.(.+)\.version\.(.*), type=gauges><>Value'
    labels:
      runtime: $1
      vendor: $2
      version: $3
    type: UNTYPED
    name: jvm_runtime_info
    help: "JVM info"

Resulting metric name: jvm_runtime{runtime="OpenJDK_Runtime_Environment",vendor="Amazon.com_Inc.",version="17.0.7+7-LTS"} 1.0

Why does "_info" get removed from the name?

[dhoard]

The relevant code is here...

https://github.com/prometheus/jmx_exporter/blob/b3823bca97c7fc2166cac7d2d12e95963b352e4b/collector/src/main/java/io/prometheus/jmx/MatchedRule.java#L73

@fstab, do you remember the specifics around why we sanitize the name?

[dhoard]

@fstab The code change was introduced as part of metric name collision work.

Do you remember the specifics? Is it necessary?

[doxsch]

We do have the same issue:

Beans:

kafka.schema.registry:type=avro-schemas-created
kafka.schema.registry:type=avro-schemas-deleted
kafka.schema.registry:type=json-schemas-created
kafka.schema.registry:type=json-schemas-deleted
kafka.schema.registry:type=protobuf-schemas-created
kafka.schema.registry:type=protobuf-schemas-deleted

Rule:

rules:
  - pattern: "kafka.schema.registry<type=(.+)-schemas-(.+)>([^:]+):"
    name: "kafka_schema_registry_schemas_$2"
    labels:
      schemaType: $1

returned metrics:

# HELP kafka_schema_registry_schemas Number of registered Avro schemas kafka.schema.registry:name=null,type=avro-schemas-created,attribute=avro-schemas-created
# TYPE kafka_schema_registry_schemas untyped
kafka_schema_registry_schemas{cluster="e1-kafka-pfnet-lab",schemaType="avro"} 0.0
kafka_schema_registry_schemas{cluster="e1-kafka-pfnet-lab",schemaType="json"} 0.0
kafka_schema_registry_schemas{cluster="e1-kafka-pfnet-lab",schemaType="protobuf"} 0.0

# HELP kafka_schema_registry_schemas_deleted Number of deleted Protobuf schemas kafka.schema.registry:name=null,type=protobuf-schemas-deleted,attribute=protobuf-schemas-deleted
# TYPE kafka_schema_registry_schemas_deleted untyped
kafka_schema_registry_schemas_deleted{cluster="e1-kafka-pfnet-lab",schemaType="avro"} 0.0
kafka_schema_registry_schemas_deleted{cluster="e1-kafka-pfnet-lab",schemaType="json"} 0.0
kafka_schema_registry_schemas_deleted{cluster="e1-kafka-pfnet-lab",schemaType="protobuf"} 0.0

The deleted metrics works but the created metric is trimmed.

[fstab]

Thanks for reporting this.

The reason is that the _info suffix is reserved for metrics of type INFO, but in the example the metric is

type: UNTYPED

Unfortunately I think type: INFO is not implemented in jmx_exporter. @dhoard would it be possible to add the INFO type? That would be the cleanest way to implement this.

[dhoard]

@fstab @iagotomas I will need to investigate the change after the pending release. The MatchedRule is used before any of the snapshot builds.

@doxsch Your issue is similar, but not easily solved since the metric is a counter, but the suffix _created is reserved for a value representing a timestamp.

[dhoard]

@fstab @zeitlinger Looking at the OpenMetrics specification...

Exposers SHOULD avoid names that could be confused with the suffixes that text format sample metric names use.

Suffixes for the respective types are:
Counter: '_total', '_created'
Summary: '_count', '_sum', '_created', '' (empty)
Histogram: '_count', '_sum', '_bucket', '_created'
GaugeHistogram: '_gcount', '_gsum', '_bucket'
Info: '_info'
Gauge: '' (empty)
StateSet: '' (empty)
Unknown: '' (empty)

Given the specifier SHOULD, the code to sanitize the name is overly strict...

https://github.com/prometheus/jmx_exporter/blob/38221102f10206b4b7a36b5cd1d9e650246aa684/collector/src/main/java/io/prometheus/jmx/MatchedRule.java#L91

The client_java library enforces the requirement, effectively changing the statement from SHOULD to MUST

I feel we need to investigate if removing the restriction in client_java is possible.

Thoughts?

[fstab]

This isn't easy, because there are many corner cases. For example, if a user registers a Gauge named my_os_info, and an Info metric named my_os, then there would be a name collision at scrape time. I think it's easier to prevent reserved suffixes at registration time so that users can be sure nothing bad can happen at scrape time.

[dhoard]

@doxsch can you create a new issue around the _created?

[fstab]

So, we just discussed this in the Prometheus Java community call, and we decided that we should support _info and _created suffixes. We'll work on a way to enable these.

[offermannu]

We have also encountered this issue and it has broken some of our Grafana dashboards. From our point of view, this is an incompatible change as it still worked with version 0.17.2. We cannot change the dashboards as otherwise the historical data would also be gone.

[dhoard]

@offermannu We are discussing possible changes.

For your rules, what metric names and types (GAUGE, COUNTER or UNTYPED) are you using that are having issues?

[dhoard]

This is a function of suffix requirements for Prometheus/OpenMetrics formats.

There is ongoing work on an OpenMetrics 2.0 proposal that will remove any suffix requirements.

[dhoard]

Removal of metric name restrictions will be resolved with:

https://github.com/prometheus/client_java/issues/1361#issuecomment-2890801832 https://github.com/prometheus/OpenMetrics/issues/286