Add or extend documentation around openmetrics integration

[lozbrown]

@mosabua

I see that in #20605 you adding docs OpenTracing but we could really do with documentation around open metrics, either similar to or extending this page https://trino.io/docs/current/admin/jmx.html

Perhaps something parallels like "Monitoring with OpenMetrics"

I'm happy to try and contribute here and can provide some examples of useful graph's etc when used with Grafana, however there are some areas i certainly don't know much about like:

In #1581 @mattstep mentions:

Trino exposes openmetrics (compatible with the prometheus scrape api) at /metrics for anything registered with jmxutils, you can also add non-jmxutils metrics via the config property here : https://github.com/airlift/airlift/blob/master/openmetrics/src/main/java/io/airlift/openmetrics/MetricsConfig.java#L34

Any documentation here or examples of doing that would be really useful. I've certainly found that not everything that is available via the jmx catalog seems to be exported to metrics endpoint. Additionally any configuration around the endpoint would be helpful.

Again documentation around this would be really helpful

[mosabua]

I am happy to review a PR .. currently I am trying to get the OpenTelemetry one done with @mattstep

If you want to provide a PR @lozbrown that would be great .. I think a separate page would be best.

[lozbrown]

@mattstep Can you help me get the right format here so I can document it?

adding the following to config.properties

openmetrics.jmx-object-names=io.trino.hdfs.s3

causes the following error

Errors:

1. Error invoking configuration method [public io.airlift.openmetrics.MetricsConfig io.airlift.openmetrics.MetricsConfig.setJmxObjectNames(java.util.List<java.lang.String>)]

if for example i was trying to access this metric provided by this jmx catalog call

select "listlocatedstatuscalls.fifteenminute.rate" from jmx.current."io.trino.hdfs.s3:name=hive,type=trinos3filesystem"

[huw0]

@lozbrown, I've found this issue while looking into enabling OpenMetrics myself and encountering the same error.

Looking at the relevant lines in...

https://github.com/airlift/airlift/blob/a30a1d15efa840632e10d7e5aba8eaa911a5c4ee/openmetrics/src/main/java/io/airlift/openmetrics/MetricsConfig.java#L41-L63

Line 44 converts the String to an ObjectName. Therefore I think that the correct format is described in the ObjectName javadoc.

I think the exception thrown in toObjectName when there is an invalid name is being caught and hidden by airlift when parsing the config?

Additionally I think the description on setJmxObjectNames is slightly ambiguous.

"JMX object names to include when retrieving all metrics

This is additional names in include in addition to the defaults. My initial expection was that this would filter rather than append.