karaf icon indicating copy to clipboard operation
karaf copied to clipboard

JMX RMI port not consistently coming online

Open mattrpav opened this issue 2 months ago • 6 comments

Possibly related exception stacktrace(s)

Exception in thread "JMX Connector Thread [service:jmx:rmi://127.0.0.1:44444/jndi/rmi://127.0.0.1:1099/karaf-root]" java.lang.RuntimeException:                                                                                             
Port already in use: 44444; 
You may have started two containers.  If you need to start a second container or the default ports are already in use update the config file etc/org.apache.karaf.management.cfg and change the Registry Port and Server Port to unused ports
	at org.apache.karaf.management.ConnectorServerFactory.lambda$init$0(ConnectorServerFactory.java:327)
	at java.base/java.lang.Thread.run(Thread.java:829)
 Exception in thread "JMX Connector Thread [service:jmx:rmi://0.0.0.0:44444/jndi/rmi://0.0.0.0:1099/karaf-root]" java.lang.RuntimeException: Could not start JMX connector server                                                            
	at org.apache.karaf.management.ConnectorServerFactory.lambda$init$0(ConnectorServerFactory.java:331)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Cannot bind to URL [rmi://0.0.0.0:1099/karaf-root]: javax.naming.NotContextException: The intermediate context karaf-root does not exist
	at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.newIOException(RMIConnectorServer.java:854)
	at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:501)
	at org.apache.karaf.management.ConnectorServerFactory.lambda$init$0(ConnectorServerFactory.java:315)
	... 1 more
Caused by: javax.naming.NotContextException: The intermediate context karaf-root does not exist
	at org.apache.xbean.naming.context.AbstractContext.lookupFinalContext(AbstractContext.java:198)
	at org.apache.xbean.naming.context.AbstractContext.addDeepBinding(AbstractContext.java:241)
	at org.apache.xbean.naming.context.AbstractContext.bind(AbstractContext.java:660)
	at org.apache.xbean.naming.context.AbstractContext.bind(AbstractContext.java:651)
	at org.apache.aries.jndi.DelegateContext.bind(DelegateContext.java:82)
	at java.naming/javax.naming.InitialContext.bind(InitialContext.java:417)
	at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.bind(RMIConnectorServer.java:713)
	at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:496)
	... 2 more

mattrpav avatar Oct 15 '25 18:10 mattrpav

Error when attempting to restart causes this as well

Exception in thread "JMX Connector Thread [service:jmx:rmi://0.0.0.0:44444/jndi/rmi://0.0.0.0:1099/karaf-root]" java.lang.RuntimeException: Could not start JMX connector server
	at org.apache.karaf.management.ConnectorServerFactory.lambda$init$0(ConnectorServerFactory.java:438)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Cannot bind to URL [rmi://0.0.0.0:1099/karaf-root]: javax.naming.NotContextException: The intermediate context karaf-root does not exist
	at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.newIOException(RMIConnectorServer.java:854)
	at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:501)
	at org.apache.karaf.management.ConnectorServerFactory.lambda$init$0(ConnectorServerFactory.java:421)
	... 1 more
Caused by: javax.naming.NotContextException: The intermediate context karaf-root does not exist
	at org.apache.xbean.naming.context.AbstractContext.lookupFinalContext(AbstractContext.java:198)
	at org.apache.xbean.naming.context.AbstractContext.addDeepBinding(AbstractContext.java:241)
	at org.apache.xbean.naming.context.AbstractContext.bind(AbstractContext.java:660)
	at org.apache.xbean.naming.context.AbstractContext.bind(AbstractContext.java:651)
	at org.apache.aries.jndi.DelegateContext.bind(DelegateContext.java:82)
	at java.naming/javax.naming.InitialContext.bind(InitialContext.java:417)
	at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.bind(RMIConnectorServer.java:713)
	at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:496)
	... 2 more
2025-11-06T00:38:25,951 | WARN | activator-1-thread-1 | Activator            | 202 - org.apache.karaf.management.server - 4.4.6 | Error destroying ConnectorServerFactory
java.rmi.NoSuchObjectException: object not exported
	at sun.rmi.transport.ObjectTable.unexportObject(ObjectTable.java:151) ~[?:?]
	at java.rmi.server.UnicastRemoteObject.unexportObject(UnicastRemoteObject.java:457) ~[?:?]
	at javax.management.remote.rmi.RMIJRMPServerImpl.unexport(RMIJRMPServerImpl.java:168) ~[?:?]
	at javax.management.remote.rmi.RMIJRMPServerImpl.closeServer(RMIJRMPServerImpl.java:233) ~[?:?]
	at javax.management.remote.rmi.RMIServerImpl.close(RMIServerImpl.java:406) ~[?:?]
	at javax.management.remote.rmi.RMIConnectorServer.stop(RMIConnectorServer.java:612) ~[?:?]
	at org.apache.karaf.management.ConnectorServerFactory.destroy(ConnectorServerFactory.java:480) ~[?:?]
	at org.apache.karaf.management.internal.Activator.doStop(Activator.java:241) ~[?:?]
	at org.apache.karaf.util.tracker.BaseActivator.run(BaseActivator.java:310) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]

mattrpav avatar Nov 13 '25 21:11 mattrpav

Seems like the port is already bound. I will check if the same logic I did for the other port is used for JMX RMI.

jbonofre avatar Nov 21 '25 12:11 jbonofre

@jbonofre the error seems to indicate that it can’t find the “karaf-root” context. Perhaps the server factory is trying to start before the management context is done being created? Or the context can’t be found from JNDI/OSGi service registry?

I’m wondering if it is a boot up race condition

mattrpav avatar Nov 21 '25 12:11 mattrpav

@mattrpav it's a consequence, the root cause is simply "Port already bound": Port already in use: 44444

That's why the context is not created.

jbonofre avatar Nov 21 '25 13:11 jbonofre

@jbonofre good catch. from the log, the Activator is running twice.

2025-11-01T03:29:38,554 | WARN  | activator-1-thread-1 | Activator                        | 203 - org.apache.karaf.management.server - 4.4.6 | java.rmi.server.hostname system property is already set to ww.xx.yy.zzz. Apache Karaf doesn't override it
2025-11-01T03:29:38,621 | WARN  | activator-1-thread-1 | Activator                        | 203 - org.apache.karaf.management.server - 4.4.6 | Error destroying ConnectorServerFactory
2025-11-01T03:29:38,622 | WARN  | activator-1-thread-1 | Activator                        | 203 - org.apache.karaf.management.server - 4.4.6 | java.rmi.server.hostname system property is already set to ww.xx.yy.zzz. Apache Karaf doesn't override it

mattrpav avatar Nov 21 '25 15:11 mattrpav

Related on first boot:

Caused by: javax.naming.NoInitialContextException

ref: https://github.com/apache/aries/blob/a5212a3b55e3abde893d3e5beb9daba5c9a7aa68/jndi/jndi-core/src/main/java/org/apache/aries/jndi/DelegateContext.java#L255

stacktrace printed to karaf console:

 Exception in thread "JMX Connector Thread [service:jmx:rmi://0.0.0.0:44444/jndi/rmi://0.0.0.0:1099/karaf-root]" java.lang.RuntimeException: Could not start JMX connector server                                                            
	at org.apache.karaf.management.ConnectorServerFactory.lambda$init$0(ConnectorServerFactory.java:331)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Cannot bind to URL [rmi://0.0.0.0:1099/karaf-root]: javax.naming.NoInitialContextException
	at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.newIOException(RMIConnectorServer.java:854)
	at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:501)
	at org.apache.karaf.management.ConnectorServerFactory.lambda$init$0(ConnectorServerFactory.java:315)
	... 1 more
Caused by: javax.naming.NoInitialContextException
	at org.apache.aries.jndi.DelegateContext.getDefaultContext(DelegateContext.java:233)
	at org.apache.aries.jndi.DelegateContext.getURLContext(DelegateContext.java:256)
	at org.apache.aries.jndi.DelegateContext.findContext(DelegateContext.java:214)
	at org.apache.aries.jndi.DelegateContext.bind(DelegateContext.java:82)
	at java.naming/javax.naming.InitialContext.bind(InitialContext.java:417)
	at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.bind(RMIConnectorServer.java:713)
	at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:496)
	... 2 more
Exception in thread "JMX Connector Thread [service:jmx:rmi://0.0.0.0:44444/jndi/rmi://0.0.0.0:1099/karaf-root]" java.lang.RuntimeException: Could not start JMX connector server
	at org.apache.karaf.management.ConnectorServerFactory.lambda$init$0(ConnectorServerFactory.java:331)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Cannot bind to URL [rmi://0.0.0.0:1099/karaf-root]: javax.naming.NoInitialContextException
	at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.newIOException(RMIConnectorServer.java:854)
	at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:501)
	at org.apache.karaf.management.ConnectorServerFactory.lambda$init$0(ConnectorServerFactory.java:315)
	... 1 more
Caused by: javax.naming.NoInitialContextException
	at org.apache.aries.jndi.DelegateContext.getDefaultContext(DelegateContext.java:233)
	at org.apache.aries.jndi.DelegateContext.getURLContext(DelegateContext.java:256)
	at org.apache.aries.jndi.DelegateContext.findContext(DelegateContext.java:214)
	at org.apache.aries.jndi.DelegateContext.bind(DelegateContext.java:82)
	at java.naming/javax.naming.InitialContext.bind(InitialContext.java:417)
	at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.bind(RMIConnectorServer.java:713)
	at java.management.rmi/javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:496)
	... 2 more

mattrpav avatar Dec 02 '25 02:12 mattrpav