JMX scrape failed: java.rmi.NoSuchObjectException: no such object in table

[AshutoshNirkhe]

trafficstars

We are running jmx-exporter as sidecar to JFROG Artifactory container in AKS cluster so as to get the visibility into the Java Beans metrics.

Extract of our current JMX config,

  config.yml: |-
    hostPort: localhost:9010
    ssl: false
    username:
    password:
    startDelaySeconds: 0
    whitelistObjectNames: ["org.jfrog.artifactory:*","Catalina:type=Connector,*","Catalina:type=ThreadPool,*","jmx.metrics:*"]
    lowercaseOutputName: true
    lowercaseOutputLabelNames: true
    rules:
      - pattern: 'Catalina<type=ThreadPool, name="(\w+-\w+)-(\d+)"><>(currentThreadCount|currentThreadsBusy|keepAliveCount|pollerThreadCount|connectionCount|running|paused|bindOnInit|keepalivetimeout|connectionTimeout|maxThreads|maxConnections|acceptCount):'
        name: catalina_threadpool_$3
        labels:
          port: "$2"
          protocol: "$1"
        help: Catalina threadpool $3
        type: GAUGE
      - pattern: ".*"

But JMX Exporter keeps failing intermittently with error "JMX scrape failed: java.rmi.NoSuchObjectException: no such object in table", resulting in 'TargetDown' Prometheus alert. Exception trace below,

SEVERE: JMX scrape failed: java.rmi.NoSuchObjectException: no such object in table
        at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283)
        at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
        at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
        at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
        at io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:94)
        at io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:468)
        at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:183)
        at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:147)
        at io.prometheus.client.CollectorRegistry.filteredMetricFamilySamples(CollectorRegistry.java:134)
        at io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:60)
        at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
        at sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)
        at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)
        at sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)
        at com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)
        at sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Mar 14, 2021 2:04:20 PM ClientCommunicatorAdmin restart
WARNING: Failed to restart: java.rmi.NoSuchObjectException: no such object in table
Mar 14, 2021 2:04:20 PM ClientCommunicatorAdmin Checker-run
WARNING: Failed to check connection: java.rmi.NoSuchObjectException: no such object in table
Mar 14, 2021 2:04:20 PM ClientCommunicatorAdmin Checker-run
WARNING: stopping

I am not sure what's the reason behind this because Artifactory itself appears to be running fine. But the metrics can be seen coming in Grafana dashboards in Gaps as below,

I couldn't find any known bugs around this apart from this generic one https://stackoverflow.com/questions/645208/java-rmi-nosuchobjectexception-no-such-object-in-table

[fstab]

To narrow this down we need to figure out if the root cause is in jmx_exporter, in the Artifactory, or in the network connection. Is there any chance that you can connect another JMX client, like Java's jconsole and see if that works?

[AshutoshNirkhe]

To narrow this down we need to figure out if the root cause is in jmx_exporter, in the Artifactory, or in the network connection. Is there any chance that you can connect another JMX client, like Java's jconsole and see if that works?

Hi @fstab Thanks for your reply and really sorry for getting back so late on this topic. The overall setup is bit weird I guess. I did try to do expose the port 9010 outside through Artifactory container via the k8s service then port-forwarded it and used jconsole on my machine to connect to localhost:9010 , but the connection doesn't work that way. But I feel this is different networking issue than what face with jmx-exporter. Any other way I can try this out or other suggestions ?

[dploeger]

@fstab I can still reproduce this here. The jmx exporter is directly installed on the same machine where the remote jmx is located.

These are the JMX settings of the application:

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.rmi.port=9911

This is the exporter config

---
hostPort: localhost:9911
ssl: false
startDelaySeconds: 0
rules:
- pattern: ".*"

Running it like that results in this Stacktrace:

Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: Sep. 20, 2022 10:21:23 VORM. io.prometheus.jmx.JmxCollector collect
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: SCHWERWIEGEND: JMX scrape failed: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.NoSuchObjectException: no such object in table]
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:370)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at java.management/javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:95)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:599)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at io.prometheus.client.Collector.collect(Collector.java:45)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:204)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:219)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:152)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at io.prometheus.client.exporter.common.TextFormat.writeOpenMetrics100(TextFormat.java:202)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at io.prometheus.client.exporter.common.TextFormat.writeFormat(TextFormat.java:57)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at io.prometheus.client.exporter.HTTPServer$HTTPMetricHandler.handle(HTTPServer.java:100)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at jdk.httpserver/sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:82)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:80)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:692)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at jdk.httpserver/com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:77)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at jdk.httpserver/sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:664)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at java.base/java.lang.Thread.run(Thread.java:834)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: Caused by: javax.naming.CommunicationException [Root exception is java.rmi.NoSuchObjectException: no such object in table]
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:137)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at java.naming/com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:207)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at java.naming/javax.naming.InitialContext.lookup(InitialContext.java:409)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1839)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at java.management.rmi/javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1813)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at java.management.rmi/javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:302)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         ... 19 more
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]: Caused by: java.rmi.NoSuchObjectException: no such object in table
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:380)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:123)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         at jdk.naming.rmi/com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:133)
Sep 20 10:21:23 ffm-teamwork-int-app-02 java[31965]:         ... 24 more

I was expecting to run it on hostPort: localhost:9010 but that doesn't work at all (connecting to the exporter times out in that case).

[dploeger]

Okay, apparently it boils down that you should use the jmx port (in my example 9010) and NOT the rmi port (9911). I've turned up the debug logs and there is a connection but somehow it's stopping midway. I'm investigating this, is probably something in the app.

[dploeger]

HA! Found it. My JNDI datasources somehow didn't respond correctly to JMX calls. I put it on the blocklist and now everything works.

So I guess the main part of this issue was, that the OP used the RMI port instead of the JMX port which lead to the error.

[dhoard]

Closed as resolved.