Active MQ Artemis / Metrics for queues like message-count, etc.

[zenhighzer]

Hi there, I am pretty new to activemq artemis, prometheus, metrics in general, etc. . so please forgive me my dumb questions :-) I successfully installed the jmx_exporter on my artemis-server and i am using the "artemis-2.yml"-example-config.

Now I can see a lot of metrics, but I hoped to see some metrics about existing queues and their message-counts, etc., Am i doing something wrong? Can I modify the config in some way to see queue-metrics?

Do you need some further informations? Looking forward for your help :-) Best Regards zen

[fstab]

jmx_exporter takes data from JMX beans and converts them to Prometheus metrics.

So the first step is to understand what data from JMX beans you are interested in. In order to figure that out, you could connect a UI-based JMX tool like jconsole or jmc (Java mission control) to your artemis process and expore what JMX data are available. Once you know what data from JMX beans you are interested in it will be easier to write a configuration for jmx_exporter to export that data to Prometheus.

[zenhighzer]

Hi Fabian, thanks for your reply. Unfortunately this sounds complicated for a simple guy like me. I dont have any clue about java, but I will have a look into tomorrow. Thank you for your help. I will give you some further feedback, best Regards!

[zenhighzer]

Hi Fabian, thanks again for your hint about jconsole. I think I found the problem, but I dont know how to solve it correctly:

• first of all: the example-config "artemis-2.yaml" in your repo is absolutely fine, there is no further configuration needed
• I was able to connect to the artemis-java-process with jconsole
• I walked throuth the MBeans-Tree "org.apache.activemq.artemis": I can see a lot of different stuff, likes addresses, queues and their attributes (e.g. MessageCount,MessagesExpired, etc.), but the attributes were all "unavailable".
• I just did a lucky shot: In artemis there is a file called management.xml, there is a section called "whitelist", which i modifed like this:

<whitelist>
         <entry domain="hawtio"/>
         <entry domain="org.apache.activemq.artemis"/>
</whitelist>

Now all the metrics are available via jconsole and jmx-exporter on port http://localhost:8080/metrics But I think this approach would disable all the other security-mechanisms in artemis, because there is some other stuff in management.xml like the following default-config-section, which would be bypassed by the whitelist-entry i mentioned before

<role-access>
         <match domain="org.apache.activemq.artemis">
            <access method="list*" roles="amq"/>
            <access method="get*" roles="amq"/>
            <access method="is*" roles="amq"/>
            <access method="set*" roles="amq"/>
            <access method="*" roles="amq"/>
        </match>

Any idea what I can do? Or should I ask the artemis-community? Thanks and Best Regards zen

[mindrunner]

@zenhighzer did you find any solutions for that?

[zenhighzer]

@mindrunner Hey, i really dont know in detail anymore but in general i configured like this... for further config-details just ask me again, i will look into the configs and give you the details:

• artemis/etc/management.xml:

<connector connector-port="1099"/>  

• artemis/etc/artemis.profile:

# Java Opts
if [ -z "$JAVA_ARGS" ]; then
    JAVA_ARGS=" -XX:+PrintClassHistogram -XX:+UseG1GC -XX:+UseStringDeduplication -Xms1G -Xmx8G -Dhawtio.disableProxy=true -Dhawtio.realm=activemq -Dhawtio.offline=true -Dhawtio.rolePrincipalClasses=org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal -Djolokia.policyLocation=${ARTEMIS_INSTANCE_ETC_URI}jolokia-access.xml -Djava.rmi.server.hostname=localhost"
fi

• jmx-exporter is running with the standalone-webserver and the "default" artemis-template from this project..

/opt/jmx_exporter/jmx-exporter-standalone.sh:

#!/usr/bin/env bash
# Script to run a java application for testing jmx4prometheus.

version=$(sed -n -e 's#.*<version>\(.*-SNAPSHOT\)</version>#\1#p' pom.xml)

# Note: You can use localhost:5556 instead of 5556 for configuring socket hostname.
java -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=5555 -jar jmx_prometheus_httpserver/target/jmx_prometheus_httpserver-${version}-jar-with-dependencies.jar 5556 artemis.yml

/opt/jmx_exporter/artemis.yml:

lowercaseOutputName: true
lowercaseOutputLabelNames: true
username: READONLY_USER_CREATED_IN_ARTEMIS
password: READONLY_USER_PASSWORD_CREATED_IN_ARTEMIS
hostPort: localhost:1099
rules:
  - pattern: "^org.apache.activemq.artemis<broker=\"([^\"]*)\"><>([^:]*):\\s(.*)"
    attrNameSnakeCase: true
    name: artemis_$2
    type: COUNTER
  - pattern: "^org.apache.activemq.artemis<broker=\"([^\"]*)\",\\s*component=addresses,\\s*address=\"([^\"]*)\"><>([^:]*):\\s(.*)"
    attrNameSnakeCase: true
    name: artemis_$3
    type: COUNTER
    labels:
        address: $2
  - pattern: "^org.apache.activemq.artemis<broker=\"([^\"]*)\",\\s*component=addresses,\\s*address=\"([^\"]*)\",\\s*subcomponent=(queue|topic)s,\\s*routing-type=\"([^\"]*)\",\\s*(queue|topic)=\"([^\"]*)\"><>([^: ]*):\\s(.*)"
    attrNameSnakeCase: true
    name: artemis_$7
    type: COUNTER
    labels:
        address: $2
        "$5": $6

I think this is was everything i did.. but i am not sure anymore... maybe this solution sucks and i am pretty sure it sucks...and maybe there is a smarter way and maybe i did something wrong... but i really dont know anything about java, artemis, and i am a little bit disappointed about the lag of documentation and support..but in the end it works and we have a beautiful dashboard in grafana with metrics about queues, message-counter, connection-counter, artemis address memory... additionally we installed the node_exporter to retrieve node-metrics like memory, cpu, disk-capacity, etc.

[zenhighzer]

[mindrunner]

Hmm, yeah. I think my setup should be quite similar. Thing is, I see metrics getting exported, but would expect more to see all stats in this dashboard: https://grafana.com/grafana/dashboards/9087

Can you try that dashboard with your setup?

Also, which dashboard are you using? Is that a homebrewed one? Are you able to share it?

[mindrunner]

Is this adding basic auth to the metrics endpoint?

[zenhighzer]

Is this adding basic auth to the metrics endpoint?

this is for connecting the the jmx-exporter to the artemis-metrics via localhost and publishing metrics on port "public"-port 5556 prometheus-server just grabs those metrics from serverip:5556 ... between prometheus-server and jmx-exporter there is no authentication

[zenhighzer]

Hmm, yeah. I think my setup should be quite similar. Thing is, I see metrics getting exported, but would expect more to see all stats in this dashboard: https://grafana.com/grafana/dashboards/9087

Can you try that dashboard with your setup?

Also, which dashboard are you using? Is that a homebrewed one? Are you able to share it?

yes, this dashboard is homebrewn. i just created it from scratch and my colleague made it more beutiful but it really was not so hard to build from scratch. getting jmx and artemis running was much much more pain. i am not sure if i can import your mentioned dashboard or if i am allowed to export our dashboard.

[mindrunner]

I think I figured it out! :) Thanks man!

[zenhighzer]

Nice dude! :-) Have you figured out any recommentations or improvements compared to "my solution"?

[mindrunner]

Yea, might be. My setup is little different though. Artemis is running in a k8s cluster, deployed by a custom helm chart as well as a custom Docker image since there is not much of a community for it as far as I can tell. The cluster has the whole Prometheus Stack stuff activated, so I was aiming for exporting the metrics on an internal ClusterIP service not exposing it to outer world. Within the cluster, I do not need any additional security for the metrics. So here is what I did:

Installed those dependencies into the environment:

ARG ACTIVEMQ_ARTEMIS_VERSION=2.19.0
ENV APACHE_IVY_VERSION=2.5.0
ENV ARTHEMIS_PROMETHEUS_METRICS_PLUGIN_VERSION=1.1.0.redhat-00001
ENV JAVAX_JSON_VERSION=1.1.4
ENV JGROUPS_KUBERNETES_VERSION=0.9.3
ENV JMX_EXPORTER_VERSION=0.16.1

jmx-exporter-config.yml

---
lowercaseOutputName: true
lowercaseOutputLabelNames: true
rules:
  - pattern: "^org.apache.activemq.artemis<broker=\"([^\"]*)\"><>([^:]*):\\s(.*)"
    attrNameSnakeCase: true
    name: artemis_$2
    type: COUNTER
  - pattern: "^org.apache.activemq.artemis<broker=\"([^\"]*)\",\\s*component=addresses,\\s*address=\"([^\"]*)\"><>([^:]*):\\s(.*)"
    attrNameSnakeCase: true
    name: artemis_$3
    type: COUNTER
    labels:
        address: $2
  - pattern: "^org.apache.activemq.artemis<broker=\"([^\"]*)\",\\s*component=addresses,\\s*address=\"([^\"]*)\",\\s*subcomponent=(queue|topic)s,\\s*routing-type=\"([^\"]*)\",\\s*(queue|topic)=\"([^\"]*)\"><>([^: ]*):\\s(.*)"
    attrNameSnakeCase: true
    name: artemis_$7
    type: COUNTER
    labels:
        address: $2
        "$5": $6

Additional Java args:

    JAVA_ARGS=" -javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent.jar=9404:/opt/jmx-exporter/etc/jmx-exporter-config.yaml -XX:+PrintClassHistogram -XX:+UseG1GC  -Dhawtio.offline=true -Dhawtio.role=admin -Dhawtio.rolePrincipalClasses=org.apache.activemq.artemis.spi.core.security.jaas.RolePrincipal -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=${JMX_PORT:-1099} -Dcom.sun.management.jmxremote.rmi.port=${JMX_RMI_PORT:-1098} -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

bootstrap.xml (Not using metrics.war here, because I do not want metrics to be exported via the default web interface which is ssl-terminated to www)

.
.
.
<web bind="http://0.0.0.0:8161" path="web">
        <app url="activemq-branding" war="activemq-branding.war"/>
        <app url="artemis-plugin" war="artemis-plugin.war"/>
        <app url="console" war="console.war"/>
      </web>
.
.
.

management.xml:

    <management-context xmlns="http://activemq.org/schema">
    <connector connector-port="1099"/>
    <authorisation>
      <allowlist>
         <entry domain="activemq"/>
         <entry domain="MyCustomLoginProvider"/>
         <entry domain="hawtio"/>
      </allowlist>
.
.
.
.

broker.xml:

.
.
.
<core>
.
.
.
        <metrics>
          <jvm-memory>true</jvm-memory>
          <jvm-gc>true</jvm-gc>
          <jvm-threads>true</jvm-threads>
          <netty-pool>false</netty-pool>
          <plugin class-name="org.apache.activemq.artemis.core.server.metrics.plugins.ArtemisPrometheusMetricsPlugin"/>
        </metrics>
</core>

I hope I mentioned everything.... The result is, prom-exporter exporting jvm-metrics as well as artemis-metric on port 9404 (unauthed), and jolokia console on port 8161 is still protected by our custom auth module.

Using a slightly modified and modernized version of this dashboard: https://grafana.com/grafana/dashboards/9087

Works perfectly fine.

[dhoard]

Closing this issue as resolved.

[Botje]

Note that adding the activemq domain to the allowlist allows anyone to connect to the specified port and issue operations such as removeAllMessages(...), addUser(...) or even forceFailover(). Be sure to protect port 1099 from malicious clients in that case.

[subhidh-agarwal]

Hi @zenhighzer Thanks for the solution and the jmx.yaml you mentioned above. One thing I noticed that all the metrics of type java.lang.String are missing - like version, name, ha_policy etc

How can that be added?

[dhoard]

@subhidh Strings that can't be converted to a numeric value will be ignored since they are not metrics.

[subhidh-agarwal]

Is there a way these can be added in some labels?

[dhoard]

I don't believe you can. I also don't see the value of scraping and storing a dummy value of 1, for every scrape.