Payara icon indicating copy to clipboard operation
Payara copied to clipboard
verified publisher icon payara

Bug Report: Exceptions when activating JMX monitoring in Payara Micro / FISH-6363

Open Absurd-Mind opened this issue 3 years ago • 1 comments

Description

Exceptions when enabling jmx-monitoring in payara micro

Expected Outcome

Enabling JMX-Monitoring should start without exceptions

Current Outcome

Exception 1:

[2022-06-15T10:57:33.041+0200] [] [←[1;93mWARNING←[0m] [] [←[1;94morg.glassfish.flashlight.datatree.impl.MethodInvokerImpl←[0m] [tid: _ThreadID=176 _ThreadName=GF Domain XML] [timeMillis: 1655283453041] [levelValue: 900] [[

java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:119)
        at java.base/java.lang.reflect.Method.invoke(Method.java:577)
        at org.glassfish.flashlight.datatree.impl.MethodInvokerImpl.getValue(MethodInvokerImpl.java:90)
        at org.glassfish.admin.monitor.StatsProviderManagerDelegateImpl.resetChildNodeStatistics(StatsProviderManagerDelegateImpl.java:594)
        at org.glassfish.admin.monitor.StatsProviderManagerDelegateImpl.resetStatistics(StatsProviderManagerDelegateImpl.java:577)
        at org.glassfish.admin.monitor.StatsProviderManagerDelegateImpl.enableStatsProvider(StatsProviderManagerDelegateImpl.java:419)
        at org.glassfish.admin.monitor.StatsProviderManagerDelegateImpl.enableStatsProviders(StatsProviderManagerDelegateImpl.java:354)
        at org.glassfish.admin.monitor.MonitoringBootstrap.handleLevelChange(MonitoringBootstrap.java:587)
        at org.glassfish.admin.monitor.MonitoringBootstrap.changed(MonitoringBootstrap.java:515)
        at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:402)
        at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:392)
        at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:281)
        at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:279)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.StringIndexOutOfBoundsException: Range [0, -1) out of bounds for length 0
        at java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:55)
        at java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:52)
        at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:213)
        at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:210)
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:98)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckFromToIndex(Preconditions.java:112)
        at java.base/jdk.internal.util.Preconditions.checkFromToIndex(Preconditions.java:349)
        at java.base/java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:1057)
        at java.base/java.lang.StringBuilder.substring(StringBuilder.java:91)
        at org.glassfish.admin.monitor.jvm.JVMRuntimeStatsProvider.getInputArguments(JVMRuntimeStatsProvider.java:137)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        ... 16 more
]]

[2022-06-15T10:57:33.053+0200] [] [←[1;93mWARNING←[0m] [] [←[1;94morg.glassfish.admin.amx.core.AMXValidator←[0m] [tid: _ThreadID=119 _ThreadName=ComplianceMonitor.ValidatorThread] [timeMillis: 1655283453053] [levelValue: 900] [[
  Attribute 'inputarguments-current' failed for amx:pp=/mon/server-mon[server],type=runtime-mon,name=jvm/runtime
java.lang.StringIndexOutOfBoundsException: Range [0, -1) out of bounds for length 0
        at java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:55)
        at java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:52)
        at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:213)
        at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:210)
        at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:98)
        at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckFromToIndex(Preconditions.java:112)
        at java.base/jdk.internal.util.Preconditions.checkFromToIndex(Preconditions.java:349)
        at java.base/java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:1057)
        at java.base/java.lang.StringBuilder.substring(StringBuilder.java:91)
        at org.glassfish.admin.monitor.jvm.JVMRuntimeStatsProvider.getInputArguments(JVMRuntimeStatsProvider.java:137)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
        at java.base/java.lang.reflect.Method.invoke(Method.java:577)
        at org.glassfish.pfl.basic.facet.FacetAccessorImpl.invoke(FacetAccessorImpl.java:131)
        at org.glassfish.gmbal.impl.MBeanImpl.invoke(MBeanImpl.java:413)
        at org.glassfish.gmbal.impl.AttributeDescriptor.get(AttributeDescriptor.java:110)
        at org.glassfish.gmbal.impl.MBeanSkeleton.getAttribute(MBeanSkeleton.java:526)
        at org.glassfish.gmbal.impl.MBeanImpl.getAttribute(MBeanImpl.java:351)
        at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:641)
        at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:679)
        at org.glassfish.admin.amx.util.jmx.MBeanProxyHandler.getAttribute(MBeanProxyHandler.java:238)
        at org.glassfish.admin.amx.core.AMXValidator._validate(AMXValidator.java:490)
        at org.glassfish.admin.amx.core.AMXValidator.validate(AMXValidator.java:1019)
        at org.glassfish.admin.amx.impl.mbean.ComplianceMonitor$ValidatorThread.doRun(ComplianceMonitor.java:226)
        at org.glassfish.admin.amx.impl.mbean.ComplianceMonitor$ValidatorThread.run(ComplianceMonitor.java:205)
]]

Exception 2:

[2022-06-15T10:59:37.212+0200] [] [←[1;93mWARNING←[0m] [NCLS-MON-00504] [←[1;94mjavax.enterprise.system.tools.monitor←[0m] [tid: _ThreadID=192 _ThreadName=GF Domain XML] [timeMillis: 1655283577212] [levelValue: 900] Encountered exception during agent attach

[2022-06-15T10:59:37.213+0200] [] [←[1;93mWARNING←[0m] [NCLS-MON-00511] [←[1;94mjavax.enterprise.system.tools.monitor←[0m] [tid: _ThreadID=192 _ThreadName=GF Domain XML] [timeMillis: 1655283577213] [levelValue: 900] [[
  Error while getting Instrumentation object from ProbeAgentMain
java.lang.ClassNotFoundException: org.glassfish.flashlight.agent.ProbeAgentMain
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        at org.glassfish.flashlight.impl.client.ProbeProviderClassFileTransformer.<clinit>(ProbeProviderClassFileTransformer.java:450)
        at org.glassfish.flashlight.impl.client.FlashlightProbeClientMediator.transformProbes(FlashlightProbeClientMediator.java:199)
        at org.glassfish.flashlight.impl.client.FlashlightProbeClientMediator.registerListener(FlashlightProbeClientMediator.java:123)
        at org.glassfish.admin.monitor.StatsProviderManagerDelegateImpl.registerStatsProviderToFlashlight(StatsProviderManagerDelegateImpl.java:670)
        at org.glassfish.admin.monitor.StatsProviderManagerDelegateImpl.enableStatsProvider(StatsProviderManagerDelegateImpl.java:424)
        at org.glassfish.admin.monitor.StatsProviderManagerDelegateImpl.enableStatsProviders(StatsProviderManagerDelegateImpl.java:354)
        at org.glassfish.admin.monitor.MonitoringBootstrap.handleLevelChange(MonitoringBootstrap.java:587)
        at org.glassfish.admin.monitor.MonitoringBootstrap.changed(MonitoringBootstrap.java:515)
        at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:402)
        at org.jvnet.hk2.config.Transactions$ConfigListenerJob.process(Transactions.java:392)
        at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:281)
        at org.jvnet.hk2.config.Transactions$ConfigListenerNotifier$1$1.call(Transactions.java:279)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
]]

[2022-06-15T10:59:37.267+0200] [] [←[1;92mINFO←[0m] [NCLS-COM-00027] [←[1;94mjavax.enterprise.system.tools.amx←[0m] [tid: _ThreadID=162 _ThreadName=RMI TCP Connection(4)-172.29.96.1] [timeMillis: 1655283577267] [levelValue: 800] AMX Attribute Change Notification for #1, Wed Jun 15 10:59:37 CEST 2022, "amx:pp=/domain/configs/config[server-config]/monitoring-service,type=module-monitoring-levels", jmx.attribute.change, HttpService: OFF => HIGH

Steps to reproduce

create a file postbootcommands: pbc.env

set-amx-enabled --enabled true --dynamic true --target server
set-jmx-monitoring-configuration --enabled true --dynamic true --amx true --logfrequency 100 --logfrequencyunit MILLISECONDS

set configs.config.server-config.monitoring-service.module-monitoring-levels.jvm=HIGH
set configs.config.server-config.monitoring-service.module-monitoring-levels.http-service=HIGH

Start payara micro: java -jar payara-micro-5.2022.2.jar --postbootcommandfile pbc.env --deploy metrics-example-1.0-SNAPSHOT.war metrics example from: https://github.com/payara/Payara-Examples/tree/master/microprofile/metrics-example

Environment

  • Distribution: Micro
  • JDK Version: OpenJDK 18
  • Operating System: Windows 11

Absurd-Mind avatar Jun 15 '22 09:06 Absurd-Mind

Hi @Absurd-Mind,

Payara Server and Payara Micro are not supported on JDK 18, you can read our supported platforms here.

I have however tested your reproducer on JDK 11 and was able to observe the same results, getting the same exceptions as you are facing, so thank you very much for your detailed reproducer. I have raised an internal issue FISH-6363 to address this bug. We always encourage you to contribute, so if you wish you can submit a PR with a fix and we will gladly review it. Otherwise, we will pick up the issue at some point in the future.

Thanks, James

JamesHillyard avatar Jun 15 '22 11:06 JamesHillyard