jmx_exporter icon indicating copy to clipboard operation
jmx_exporter copied to clipboard

Published 20 hours ago • verified publisher icon prometheus

Jmx exporter doesn't work with exporter-exporter

Open DAtanassov1 opened this issue 3 years ago • 16 comments

Jmx exporter doesn't work with exporter-exporter. Exporter-exporter expects the following html page for the proxy request "http://localhost:/proxy?module=jmx" but receive the /metrics

<html>
<head><title>Jmx Exporter</title></head>
<body>
<h1>Jmx Exporter</h1>
<p><a href="/metrics">Metrics</a></p>
</body>
</html>

DAtanassov1 avatar Jul 18 '21 01:07 DAtanassov1

Hi, please let me know your exporter_exporter configuration, the jmx_exporter configuration, and the command line you used to start your Java application. I'll have a look and try to figure out what's wrong.

fstab avatar Jul 18 '21 19:07 fstab

Hi,

Here is the exporter-exporter config:

{
  "service":
    {
        "name": "jmx_exporter",
        "tags": ["jmx_exporter"],
        "address": "",
        "port": 9999,
        "check": {
            "id": "jmx_exporter_status",
            "name": "Jmx exporter Status",
            "tcp": "localhost:9101",
            "interval": "30s",
            "timeout": "10s"
        }
    }
}

The jmx config file is almost the same as this one: https://github.com/prometheus/jmx_exporter/blob/master/example_configs/activemq.yml Anyway, I've attached it here (activemq.yml) activemq.txt

The command line to run the Java application is part of the ActiveMQ service start command:

ACTIVEMQ_OPTS="$ACTIVEMQ_OPTS $ACTIVEMQ_SUNJMX_START $ACTIVEMQ_SSL_OPTS -Djava.awt.headless=true -Djava.io.tmpdir=\"${ACTIVEMQ_TMP}\" -javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent-0.16.1.jar=9101:/opt/jmx-exporter/activemq.yml"

DAtanassov1 avatar Jul 18 '21 19:07 DAtanassov1

Hi,

These are the error messages I am getting when I run exporter-exporter in debug mode. See the attached file. jmx_errors.log

DAtanassov1 avatar Jul 18 '21 19:07 DAtanassov1

The error in jmx_errors.log says:

Verification for module 'jmx' failed: Failed to decode metrics from proxied server: text format parsing error in line 34: second TYPE line for metric name "jvm_classes_loaded", or TYPE reported after samples

I don't see where this comes from. I ran an example, and the jvm_classes_loaded and jvm_classes_loaded_total metrics look correct:

# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_loaded gauge
jvm_classes_loaded 2536.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 2536.0

Please run curl http://localhost:9101/metrics on the machine where ActiveMQ with the Java agent is running and check the jvm_classes_loaded and jvm_classes_loaded_total metrics. Do they look like the example above, or is there anything wrong with the TYPE?

fstab avatar Jul 19 '21 21:07 fstab

I think I might see what's going on. That's valid Prometheus format, but could run into issues if you've something that's treating it as OpenMetrics. I'd suggest remaning the two counters to loads and unloads to avoid the clash, though I'm on unclear on what exactly is noticing the clash.

brian-brazil avatar Jul 19 '21 21:07 brian-brazil

Fabian and Brian,

Yes, I am getting the same output from curl command:

# HELP jvm_threads_state Current count of threads by state
# TYPE jvm_threads_state gauge
jvm_threads_state{state="RUNNABLE",} 44.0
jvm_threads_state{state="NEW",} 0.0
jvm_threads_state{state="BLOCKED",} 0.0
jvm_threads_state{state="TIMED_WAITING",} 46.0
jvm_threads_state{state="WAITING",} 8.0
jvm_threads_state{state="TERMINATED",} 0.0
# HELP jvm_classes_loaded The number of classes that are currently loaded in the JVM
# TYPE jvm_classes_loaded gauge
jvm_classes_loaded 7730.0
# HELP jvm_classes_loaded_total The total number of classes that have been loaded since the JVM has started execution
# TYPE jvm_classes_loaded_total counter
jvm_classes_loaded_total 7821.0
# HELP jvm_classes_unloaded_total The total number of classes that have been unloaded since the JVM has started execution
# TYPE jvm_classes_unloaded_total counter
jvm_classes_unloaded_total 91.0
# HELP jvm_info VM version info
# TYPE jvm_info gauge
jvm_info{runtime="OpenJDK Runtime Environment",vendor="Red Hat, Inc.",version="16.0.1+9",} 1.0

The issue is not with the metrics output itself. The issue is that exporter-exporter expects an HTML page to this request "http://localhost:/proxy?module=jmx". The expected HTML page I've posted in my first message. The error logs I've posted are from exporter-exporter. The HTTP error is 500 reported by exporter-exporter.

I am trying to setup jmx-exporter to be directly queried from Prometheus, but so far I am facing other issues

DAtanassov1 avatar Jul 19 '21 22:07 DAtanassov1

Stupid question, but I want to make sure that we are looking at the same thing: Are you talking about https://github.com/QubitProducts/exporter_exporter? I'm wondering because the config you posted above (with keys like check) does not fit with the config described on that project's README.md (I don't find check there).

fstab avatar Jul 20 '21 18:07 fstab

The config I posted is for consul service, so the jmx-exporter service is registered through the exporter-exporter service. The config for the jmx-exporter in exporter-exporter is very simple:

method: http
http:
   port: 9101

DAtanassov1 avatar Jul 20 '21 22:07 DAtanassov1

I'm still struggling to reproduce this. Here are the commands I used trying to set this up:

Download, unpack, create config:

#!/bin/bash

curl -OL http://archive.apache.org/dist/activemq/5.16.2/apache-activemq-5.16.2-bin.tar.gz
curl -OL https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.16.1/jmx_prometheus_javaagent-0.16.1.jar
curl -OL https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/activemq.yml
curl -OL https://github.com/QubitProducts/exporter_exporter/releases/download/v0.4.2/exporter_exporter-0.4.2.linux-amd64.tar.gz

tar xfz apache-activemq-5.16.2-bin.tar.gz
tar xfz exporter_exporter-0.4.2.linux-amd64.tar.gz

cat > expexp.yaml <<EOF
modules:
  jmx:
    method: http
    http:
      port: 9101
EOF

Run ActiveMQ

ACTIVEMQ_OPTS=-javaagent:./jmx_prometheus_javaagent-0.16.1.jar=9101:activemq.yml ./apache-activemq-5.16.2/bin/activemq console

Run exporter_exporter

./exporter_exporter-0.4.2.linux-amd64/exporter_exporter -config.file ./expexp.yaml

Metrics on http://localhost:9999/proxy?module=jmx are as expected. Can you spot the difference between my setup and your's?

fstab avatar Jul 21 '21 21:07 fstab

I have one difference - I run exporter-exporter with TLS: -web.tls.listen-address=:9999 -web.tls.cert=node_cert.pem -web.tls.key=key.pem -web.tls.ca=prometheus_cert.pem That is not a problem. I still get correct metrics locally with: curl https://localhost:9999/proxy?module=jmx or remotely: curl https://<host_ip>:9999/proxy?module=jmx I am getting the correct JMX metrics even when I click on the endpoint showing as DOWN in the attached image. The issue is that the Prometheus server is getting an http 500 error when it is trying to check/verify the service.

The JMX exporter works fine and the endpoint in Prometheus is OK if I expose the 9101 port directly to Prometheus. jmx

DAtanassov1 avatar Jul 22 '21 02:07 DAtanassov1

Opened an issue here https://github.com/prometheus/common/issues/319

fstab avatar Jul 25 '21 11:07 fstab

I renamed jvm_classes_loaded to jvm_classes_currently_loaded. This will solve the issue. It will be rolled out with the next release. See https://github.com/prometheus/client_java/pull/681

fstab avatar Aug 06 '21 22:08 fstab

I renamed jvm_classes_loaded to jvm_classes_currently_loaded. This will solve the issue. It will be rolled out with the next release. See prometheus/client_java#681

@fstab are there plans for a release of the jmx_exporter containing this fix?

peterschrott avatar Apr 05 '22 14:04 peterschrott

I've created a TCP dump with exporter-exporter and here is what I see: https://gist.github.com/marthydavid/dfb07dcdb891351f0cea33e94bcc1f38 exporter-exporter or prometheus i cannot tell uses different Accept header:

'accept: application/openmetrics-text'

The important part is:

# TYPE jvm info
# HELP jvm VM version info 
jvm_info{runtime="OpenJDK Runtime Environment",vendor="Red Hat, Inc.",version="1.8.0_292-b10"} 1.0

Maybe in this case it should show TYPE summary or TYPE count not info

exporter-exporter journal output:

Sep 27 15:25:43 myhost exporter_exporter[91007]: time="2022-09-27T15:25:43+02:00" level=error msg="Verification for module 'connector-exporter' failed: Failed to decode metrics from proxied server: text format parsing error in line 990: unknown metric type \"info\""

Older jmx exporter uses gauge:

# TYPE jvm_info gauge
jvm_info{version="1.8.0_292-b10",vendor="Red Hat, Inc.",runtime="OpenJDK Runtime Environment",} 1.0
# HELP jmx_exporter_build_info A metric with a constant '1' value labeled with the version of the JMX exporter.
# TYPE jmx_exporter_build_info gauge
jmx_exporter_build_info{version="0.12.1-SNAPSHOT",name="jmx_prometheus_javaagent",} 1.0

marthydavid avatar Sep 27 '22 13:09 marthydavid

The 0.12.1-SNAPSHOT uses cae9f262557

marthydavid avatar Sep 27 '22 13:09 marthydavid

Easy check:

[root@myhost ~]# curl -s -H 'Accept: application/text' localhost:9405/metrics|grep 'jvm.*info'
# HELP jvm_info VM version info
# TYPE jvm_info gauge
jvm_info{runtime="OpenJDK Runtime Environment",vendor="Red Hat, Inc.",version="1.8.0_292-b10",} 1.0
[root@myhost ~]# curl -s -H 'Accept: application/openmetrics-text' localhost:9405/metrics|grep 'jvm.*info'
# TYPE jvm info
# HELP jvm VM version info
jvm_info{runtime="OpenJDK Runtime Environment",vendor="Red Hat, Inc.",version="1.8.0_292-b10"} 1.0

marthydavid avatar Sep 27 '22 13:09 marthydavid