jmx_exporter
jmx_exporter copied to clipboard
FATAL ERROR in native method: processing of -javaagent failed
I'm having issues deploying our sink connector with the JMX Exporter agent in a kubernetes environment. Regardless of what config I use, whether my own, the example config or just a very stripped down config. I continuously get the same error
Exception in thread "main" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
Caused by: java.lang.NoSuchFieldError: UNKNOWN
at io.prometheus.jmx.JmxCollector$Rule.<init>(JmxCollector.java:59)
at io.prometheus.jmx.JmxCollector.loadConfig(JmxCollector.java:216)
at io.prometheus.jmx.JmxCollector.<init>(JmxCollector.java:93)
at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:29)
... 6 more
*** java.lang.instrument ASSERTION FAILED ***: "result" with message agent load/premain call failed at src/java.instrument/share/native/libinstrument/JPLISAgent.c line: 422
FATAL ERROR in native method: processing of -javaagent failed, processJavaStart failed
The current config I'm using is jms_exporter.yaml
---
rules:
- pattern: ".*"
My dockerfile is as follow
## SETUP ##
FROM maven:3.8.6-openjdk-18-slim as maven_dependencies
WORKDIR /app
COPY pom.xml .
RUN --mount=type=cache,target=/root/.m2 mvn dependency:copy-dependencies -DoutputDirectory=libs -Dmaven.test.skip
## RUN ##
FROM confluentinc/cp-kafka-connect:7.2.1
ARG ENVIRONMENT
ARG URL_ENV
ARG CONNECT_GROUP_ID
ENV env=$ENVIRONMENT
ENV url_env=$URL_ENV
ENV CONNECT_GROUP_ID = CONNECT_GROUP_ID
ENV KAFKA_OPTS="-javaagent:/usr/share/java/cp-base-new/jmx_prometheus_javaagent-0.18.0.jar=5556:/opt/prometheus/jmx_exporter.yaml"
# Copy Maven dependencies
COPY --from=maven_dependencies /app/libs /usr/share/java/cp-base-new/
# Copy JMX Configs
COPY jmx_exporter.yaml /opt/prometheus/jmx_exporter.yaml
# Copy init script
COPY ./init.sh init.sh
# Install Mongo Connect
RUN confluent-hub install --no-prompt mongodb/kafka-connect-mongodb:1.7.0
USER root
RUN chown appuser init.sh
RUN chown appuser /opt/prometheus
RUN chmod +x init.sh
USER appuser
CMD ["sh", "-c", "./init.sh $env $CONNECT_GROUP_ID"]
The shell script just starts the kafka connector and pushes configs to the connector.
@Lucky38i This appears to be some type of mismatch between jars in the image.
The first step would be to run the container without installing the connector to rule in/out the connector.
Some initial code analysis ...
Caused by: java.lang.NoSuchFieldError: UNKNOWN
at io.prometheus.jmx.JmxCollector$Rule.<init>(JmxCollector.java:59)
Is caused when code is referencing a class field or enum and it doesn't exist in the referenced class.
static class Rule {
Pattern pattern;
String name;
String value;
Double valueFactor = 1.0;
String help;
boolean attrNameSnakeCase;
boolean cache = false;
Type type = Type.UNKNOWN; // <- reference to Type.UNKNOWN
ArrayList<String> labelNames;
ArrayList<String> labelValues;
}
The Type
enum is defined in the Collector
class from the client_java
project dependency ...
public enum Type {
UNKNOWN, // This is untyped in Prometheus text format.
COUNTER,
GAUGE,
STATE_SET,
INFO,
HISTOGRAM,
GAUGE_HISTOGRAM,
SUMMARY,
}
https://github.com/prometheus/jmx_exporter/blob/main/collector/src/main/java/io/prometheus/jmx/JmxCollector.java
Type.UNKNOWN
was added to client_java
version 0.16.0
. If there are any jars that use older versions of the code or an older version of the jar is in the classpath, this error could occur.
@Lucky38i I believe I see the issue.
ENV KAFKA_OPTS="-javaagent:/usr/share/java/cp-base-new/jmx_prometheus_javaagent-0.18.0.jar=5556:/opt/prometheus/jmx_exporter.yaml"
... is adding jmx_prometheus_javaagent-0.18.0.jar
to the Connect classpath, which is incorrect.
Please test putting the exporter jar in a different directory not in the Connect classpath.
@Lucky38i I believe I see the issue.
ENV KAFKA_OPTS="-javaagent:/usr/share/java/cp-base-new/jmx_prometheus_javaagent-0.18.0.jar=5556:/opt/prometheus/jmx_exporter.yaml"
... is adding
jmx_prometheus_javaagent-0.18.0.jar
to the Connect classpath, which is incorrect.Please test putting the exporter jar in a different directory not in the Connect classpath.
I have this exact issue. I've put the jar file in a different directory yet the issue exists.
KAFKA_OPTS=-javaagent:/opt/prometheus/jmx-exporter-0.20.0.jar=1234:/opt/prometheus/kafka-jmx-exporter.yml
@Marchosiax can you publish a Docker image I can use for testing?
@dhoard It's a standard config I put the config files here so you can run locally
docker-compose.yml
version: '3.8'
services:
zookeeper:
image: confluentinc/cp-zookeeper:7.1.1
hostname: zookeeper
volumes:
- zookeeper-data:/var/lib/zookeeper/data
- zookeeper-log:/var/lib/zookeeper/log
environment:
- ALLOW_ANONYMOUS_LOGIN=yes
- ZOOKEEPER_CLIENT_PORT=2181
networks:
- default
deploy:
restart_policy:
condition: on-failure
kafka-1:
build: docker-images/kafka
hostname: kafka-1
volumes:
- kafka-1:/var/lib/kafka/data
environment:
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
- ALLOW_PLAINTEXT_LISTENER=yes
- KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
- KAFKA_LISTENERS=CLIENT://kafka-1:29092,EXTERNAL://kafka-1:9092
- KAFKA_ADVERTISED_LISTENERS=CLIENT://kafka-1:29092,EXTERNAL://kafka-1:9092
- KAFKA_INTER_BROKER_LISTENER_NAME=CLIENT
- KAFKA_UNCLEAN_LEADER_ELECTION_ENABLE=false
- KAFKA_OPTS=-javaagent:/opt/prometheus/jmx-exporter-0.20.0.jar=1234:/opt/prometheus/kafka-jmx-exporter.yml
depends_on:
- zookeeper
networks:
- default
deploy:
restart_policy:
condition: on-failure
kafka-2:...
kafka-3:...
./docker-images/kafka/Dockerfile
exporter jar file and and exporter config file are in this folder
FROM confluentinc/cp-kafka:7.1.1
USER root
RUN mkdir /opt/prometheus
RUN chmod +rx /opt/prometheus
COPY jmx-exporter-0.20.0.jar /opt/prometheus
COPY kafka-jmx-exporter.yml /opt/prometheus
@Marchosiax while you are adding the new JMX exporter, you haven't removed the version (/usr/share/java/cp-base-new/jmx_prometheus_javaagent-0.14.0.jar
) shipped in the base image
I believe /usr/share/java/cp-base-new/
is in the standard classpath, which will causes issues.
I would try to remove the version shipped in the base image (/usr/share/java/cp-base-new/jmx_prometheus_javaagent-0.14.0.jar
)
Try this one. It worked for me:
kafka:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- 29092:29092
hostname: kafka
volumes:
- /home/kafka/jmx-exporter/:/usr/share/jmx-exporter
- /home/kafka/jmx-exporter/jmx_prometheus_javaagent-0.12.0.jar:/usr/share/java/cp-base-new/jmx_prometheus_javaagent-0.12.0.jar
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
KAFKA_JMX_OPTS: -javaagent:/usr/share/jmx-exporter/jmx_prometheus_javaagent-0.12.0.jar=1234:/usr/share/jmx-exporter/kafka-broker.yml -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false```