jmx_exporter
jmx_exporter copied to clipboard
Published
20 hours ago •
prometheus
prometheus
glassfish 7.0.20 fails to start
Tested today the official image provided by ee4j for glassfish with jmx_exporter.
I tried to add the files but was not allowed, so I paste them here:
cat Containerfile
FROM ghcr.io/eclipse-ee4j/glassfish:7.0.20
COPY config.yaml /opt/
COPY jmx_prometheus_javaagent-1.1.0.jar /opt/
RUN true \
&& AS_START_TIMEOUT=120000 asadmin start-domain \
&& curl -o /dev/null http://localhost:4848 \
&& asadmin create-jvm-options "-javaagent\:/opt/jmx_prometheus_javaagent-1.1.0.jar=9090\:/opt/config.yaml" \
&& asadmin stop-domain --kill
cat config.yaml:
rules:
- pattern: ".*"
I build the image with podman build -t bug and started the image:
2025-01-28 15:44:23.425 | main | INFO | io.prometheus.jmx.JavaAgent | Starting ...
2025-01-28 15:44:23.541 | main | INFO | io.prometheus.jmx.JavaAgent | HTTP enabled [true]
2025-01-28 15:44:23.541 | main | INFO | io.prometheus.jmx.JavaAgent | HTTP host:port [0.0.0.0:9090]
2025-01-28 15:44:23.541 | main | INFO | io.prometheus.jmx.JavaAgent | OpenTelemetry enabled [false]
2025-01-28 15:44:23.558 | main | INFO | io.prometheus.jmx.JavaAgent | Running ...
Launching GlassFish on Felix platform
OSGI framework packages:
org.glassfish.main.jul;uses:="org.glassfish.main.jul.cfg,org.glassfish.main.jul.handler";version="7.0.20",org.glassfish.main.jul.cfg;version="7.0.20",org.glassfish.main.jul.env;version="7.0.20",org.glassfish.main.jul.formatter;uses:="org.glassfish.main.jul.cfg,org.glassfish.main.jul.record";version="7.0.20",org.glassfish.main.jul.handler;uses:="org.glassfish.main.jul.cfg,org.glassfish.main.jul.formatter,org.glassfish.main.jul.record";version="7.0.20",org.glassfish.main.jul.record;version="7.0.20",org.glassfish.main.jul.rotation;version="7.0.20",org.glassfish.main.jul.tracing;version="7.0.20", org.osgi.dto;version="1.1.1",org.osgi.framework;version="1.10",org.osgi.framework.connect;uses:="org.osgi.framework,org.osgi.framework.launch";version="1.0",org.osgi.framework.dto;uses:="org.osgi.dto";version="1.8",org.osgi.framework.hooks.bundle;uses:="org.osgi.framework";version="1.1",org.osgi.framework.hooks.resolver;uses:="org.osgi.framework.wiring";version="1.0",org.osgi.framework.hooks.service;uses:="org.osgi.framework";version="1.1",org.osgi.framework.hooks.weaving;uses:="org.osgi.framework.wiring";version="1.1",org.osgi.framework.launch;uses:="org.osgi.framework";version="1.2",org.osgi.framework.namespace;uses:="org.osgi.resource";version="1.2",org.osgi.framework.startlevel;uses:="org.osgi.framework";version="1.0",org.osgi.framework.startlevel.dto;uses:="org.osgi.dto";version="1.0",org.osgi.framework.wiring;uses:="org.osgi.framework,org.osgi.resource";version="1.2",org.osgi.framework.wiring.dto;uses:="org.osgi.dto,org.osgi.resource.dto";version="1.3",org.osgi.resource;version="1.0.1",org.osgi.resource.dto;uses:="org.osgi.dto";version="1.0.1",org.osgi.service.condition;version="1.0",org.osgi.service.packageadmin;uses:="org.osgi.framework";version="1.2.1",org.osgi.service.resolver;uses:="org.osgi.resource";version="1.1.1",org.osgi.service.startlevel;uses:="org.osgi.framework";version="1.1.1",org.osgi.service.url;version="1.0.1",org.osgi.util.tracker;uses:="org.osgi.framework";version="1.5.3", org.glassfish.embeddable;version="7.0.20",org.glassfish.embeddable.spi;uses:="org.glassfish.embeddable";version="7.0.20", org.glassfish.main.jul;uses:="org.glassfish.main.jul.cfg,org.glassfish.main.jul.handler";version="7.0.20",org.glassfish.main.jul.cfg;version="7.0.20",org.glassfish.main.jul.env;version="7.0.20",org.glassfish.main.jul.formatter;uses:="org.glassfish.main.jul.cfg,org.glassfish.main.jul.record";version="7.0.20",org.glassfish.main.jul.handler;uses:="org.glassfish.main.jul.cfg,org.glassfish.main.jul.formatter,org.glassfish.main.jul.record";version="7.0.20",org.glassfish.main.jul.record;version="7.0.20",org.glassfish.main.jul.rotation;version="7.0.20",org.glassfish.main.jul.tracing;version="7.0.20"
JDK provided packages:
com.sun.jarsigner, com.sun.java.accessibility.util, com.sun.jdi, com.sun.jdi.connect, com.sun.jdi.connect.spi, com.sun.jdi.event, com.sun.jdi.request, com.sun.management, com.sun.net.httpserver, com.sun.net.httpserver.spi, com.sun.nio.file, com.sun.nio.sctp, com.sun.security.auth, com.sun.security.auth.callback, com.sun.security.auth.login, com.sun.security.auth.module, com.sun.security.jgss, com.sun.source.doctree, com.sun.source.tree, com.sun.source.util, com.sun.tools.attach, com.sun.tools.attach.spi, com.sun.tools.javac, com.sun.tools.jconsole, java.applet, java.awt, java.awt.color, java.awt.datatransfer, java.awt.desktop, java.awt.dnd, java.awt.event, java.awt.font, java.awt.geom, java.awt.im, java.awt.im.spi, java.awt.image, java.awt.image.renderable, java.awt.print, java.beans, java.beans.beancontext, java.io, java.lang, java.lang.annotation, java.lang.constant, java.lang.instrument, java.lang.invoke, java.lang.management, java.lang.module, java.lang.ref, java.lang.reflect, java.lang.runtime, java.math, java.net, java.net.http, java.net.spi, java.nio, java.nio.channels, java.nio.channels.spi, java.nio.charset, java.nio.charset.spi, java.nio.file, java.nio.file.attribute, java.nio.file.spi, java.rmi, java.rmi.dgc, java.rmi.registry, java.rmi.server, java.security, java.security.cert, java.security.interfaces, java.security.spec, java.sql, java.text, java.text.spi, java.time, java.time.chrono, java.time.format, java.time.temporal, java.time.zone, java.util, java.util.concurrent, java.util.concurrent.atomic, java.util.concurrent.locks, java.util.function, java.util.jar, java.util.logging, java.util.prefs, java.util.random, java.util.regex, java.util.spi, java.util.stream, java.util.zip, javax.accessibility, javax.annotation.processing, javax.crypto, javax.crypto.interfaces, javax.crypto.spec, javax.imageio, javax.imageio.event, javax.imageio.metadata, javax.imageio.plugins.bmp, javax.imageio.plugins.jpeg, javax.imageio.plugins.tiff, javax.imageio.spi, javax.imageio.stream, javax.lang.model, javax.lang.model.element, javax.lang.model.type, javax.lang.model.util, javax.management, javax.management.loading, javax.management.modelmbean, javax.management.monitor, javax.management.openmbean, javax.management.relation, javax.management.remote, javax.management.remote.rmi, javax.management.timer, javax.naming, javax.naming.directory, javax.naming.event, javax.naming.ldap, javax.naming.ldap.spi, javax.naming.spi, javax.net, javax.net.ssl, javax.print, javax.print.attribute, javax.print.attribute.standard, javax.print.event, javax.rmi.ssl, javax.script, javax.security.auth, javax.security.auth.callback, javax.security.auth.kerberos, javax.security.auth.login, javax.security.auth.spi, javax.security.auth.x500, javax.security.cert, javax.security.sasl, javax.smartcardio, javax.sound.midi, javax.sound.midi.spi, javax.sound.sampled, javax.sound.sampled.spi, javax.sql, javax.sql.rowset, javax.sql.rowset.serial, javax.sql.rowset.spi, javax.swing, javax.swing.border, javax.swing.colorchooser, javax.swing.event, javax.swing.filechooser, javax.swing.plaf, javax.swing.plaf.basic, javax.swing.plaf.metal, javax.swing.plaf.multi, javax.swing.plaf.nimbus, javax.swing.plaf.synth, javax.swing.table, javax.swing.text, javax.swing.text.html, javax.swing.text.html.parser, javax.swing.text.rtf, javax.swing.tree, javax.swing.undo, javax.tools, javax.transaction.xa, javax.xml, javax.xml.catalog, javax.xml.crypto, javax.xml.crypto.dom, javax.xml.crypto.dsig, javax.xml.crypto.dsig.dom, javax.xml.crypto.dsig.keyinfo, javax.xml.crypto.dsig.spec, javax.xml.datatype, javax.xml.namespace, javax.xml.parsers, javax.xml.stream, javax.xml.stream.events, javax.xml.stream.util, javax.xml.transform, javax.xml.transform.dom, javax.xml.transform.sax, javax.xml.transform.stax, javax.xml.transform.stream, javax.xml.validation, javax.xml.xpath, jdk.dynalink, jdk.dynalink.beans, jdk.dynalink.linker, jdk.dynalink.linker.support, jdk.dynalink.support, jdk.javadoc.doclet, jdk.jfr, jdk.jfr.consumer, jdk.jshell, jdk.jshell.execution, jdk.jshell.spi, jdk.jshell.tool, jdk.management.jfr, jdk.net, jdk.nio, jdk.nio.mapmode, jdk.security.jarsigner, jdk.swing.interop, netscape.javascript, org.ietf.jgss, org.w3c.dom, org.w3c.dom.bootstrap, org.w3c.dom.css, org.w3c.dom.events, org.w3c.dom.html, org.w3c.dom.ls, org.w3c.dom.ranges, org.w3c.dom.stylesheets, org.w3c.dom.traversal, org.w3c.dom.views, org.w3c.dom.xpath, org.xml.sax, org.xml.sax.ext, org.xml.sax.helpers, sun.misc, sun.reflect
Jan 28, 2025 3:44:25 PM com.sun.enterprise.server.logging.LogManagerService postConstruct
INFO: Detected other than GlassFishLogManager, the LogManagerService's features may be limited. Used log manager: java.util.logging.LogManager@7acac06a
Jan 28, 2025 3:44:25 PM com.sun.enterprise.server.logging.LogManagerService reconfigure
INFO: Using property file: /opt/glassfish7/glassfish/domains/domain1/config/logging.properties
Can't load log handler "org.glassfish.main.jul.handler.SimpleLogHandler"
java.lang.ClassNotFoundException: org.glassfish.main.jul.handler.SimpleLogHandler
java.lang.ClassNotFoundException: org.glassfish.main.jul.handler.SimpleLogHandler
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
at java.logging/java.util.logging.LogManager.createLoggerHandlers(LogManager.java:1005)
at java.logging/java.util.logging.LogManager$4.run(LogManager.java:975)
at java.logging/java.util.logging.LogManager$4.run(LogManager.java:971)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
at java.logging/java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:971)
at java.logging/java.util.logging.LogManager.initializeGlobalHandlers(LogManager.java:2424)
at java.logging/java.util.logging.LogManager$RootLogger.accessCheckedHandlers(LogManager.java:2526)
at java.logging/java.util.logging.Logger.getHandlers(Logger.java:2090)
at java.logging/java.util.logging.Logger.log(Logger.java:977)
at java.logging/java.util.logging.Logger.doLog(Logger.java:1007)
at java.logging/java.util.logging.Logger.log(Logger.java:1073)
at com.sun.enterprise.server.logging.LogManagerService.postConstruct(LogManagerService.java:127)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:362)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:410)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:479)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:288)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2111)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:68)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1366)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1294)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Can't load log handler "org.glassfish.main.jul.handler.GlassFishLogHandler"
java.lang.ClassNotFoundException: org.glassfish.main.jul.handler.GlassFishLogHandler
java.lang.ClassNotFoundException: org.glassfish.main.jul.handler.GlassFishLogHandler
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
at java.logging/java.util.logging.LogManager.createLoggerHandlers(LogManager.java:1005)
at java.logging/java.util.logging.LogManager$4.run(LogManager.java:975)
at java.logging/java.util.logging.LogManager$4.run(LogManager.java:971)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
at java.logging/java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:971)
at java.logging/java.util.logging.LogManager.initializeGlobalHandlers(LogManager.java:2424)
at java.logging/java.util.logging.LogManager$RootLogger.accessCheckedHandlers(LogManager.java:2526)
at java.logging/java.util.logging.Logger.getHandlers(Logger.java:2090)
at java.logging/java.util.logging.Logger.log(Logger.java:977)
at java.logging/java.util.logging.Logger.doLog(Logger.java:1007)
at java.logging/java.util.logging.Logger.log(Logger.java:1073)
at com.sun.enterprise.server.logging.LogManagerService.postConstruct(LogManagerService.java:127)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:362)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:410)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:479)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:288)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2111)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:68)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1366)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1294)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Can't load log handler "org.glassfish.main.jul.handler.SyslogHandler"
java.lang.ClassNotFoundException: org.glassfish.main.jul.handler.SyslogHandler
java.lang.ClassNotFoundException: org.glassfish.main.jul.handler.SyslogHandler
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
at java.logging/java.util.logging.LogManager.createLoggerHandlers(LogManager.java:1005)
at java.logging/java.util.logging.LogManager$4.run(LogManager.java:975)
at java.logging/java.util.logging.LogManager$4.run(LogManager.java:971)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)
at java.logging/java.util.logging.LogManager.loadLoggerHandlers(LogManager.java:971)
at java.logging/java.util.logging.LogManager.initializeGlobalHandlers(LogManager.java:2424)
at java.logging/java.util.logging.LogManager$RootLogger.accessCheckedHandlers(LogManager.java:2526)
at java.logging/java.util.logging.Logger.getHandlers(Logger.java:2090)
at java.logging/java.util.logging.Logger.log(Logger.java:977)
at java.logging/java.util.logging.Logger.doLog(Logger.java:1007)
at java.logging/java.util.logging.Logger.log(Logger.java:1073)
at com.sun.enterprise.server.logging.LogManagerService.postConstruct(LogManagerService.java:127)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1268)
at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:362)
at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:410)
at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:479)
at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:288)
at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:65)
at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2111)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:68)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1366)
at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1294)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
SSLParams =org.glassfish.admin.mbeanserver.ssl.SSLParams@47c50497
SSLParams =org.glassfish.admin.mbeanserver.ssl.SSLParams@47c50497
Needless to say everything works as expected WITHOUT the java property for the agent. TBH, I have no idea where to start?
@rlfnb The issue is that the JMX Exporter Java Agent runs in a different classloader than the application and can't find the required glassfish logging jars.
This appears similar to JMX Exporter issues...
- https://github.com/prometheus/jmx_exporter/issues/455
- https://github.com/prometheus/jmx_exporter/issues/976
- https://github.com/prometheus/jmx_exporter/issues/1000
... and GlashFish issues...
- https://github.com/eclipse-ee4j/glassfish/issues/25133
- https://github.com/eclipse-ee4j/glassfish/pull/25183
Background
The JMX Exporter Java Agent uses standard Java logging. Because standard Java logging is loaded as part of the Java agent before GlassFish, the wrong Java logging implementation is loaded.
Typically, you would need to find all of the GlassFish jars and add them to the boot classpath, along with defining a Java system property to define the LogManager implementation. (reference: https://github.com/prometheus/jmx_exporter/issues/455)
However, per the comment by @dmatej in GlassFish issue https://github.com/eclipse-ee4j/glassfish/issues/25133, I'm not sure there is a viable configuration workaround.
We may need to revisit logging in the JMX Exporter.
cc @fstab
Hi, I guess there are two possible workarounds:
- Avoid using Java Util Logging in agent which I believe is not useful for you
- Explicitly start glassfish with the JDK's LogManager set using the JVM option. Then GlassFish will lose some features, but it should start at least.
We are now working on 7.1.0 where it should be resolved. I hope I will not forget this to check.
