1.32.1: java.lang.IllegalStateException: agent initializer should be loaded in boot loader

[macau23]

Describe the bug

Under WebLogic with jdk-11.0.15.1 I am using these extra arguments:

-javaagent:/tmp/opentelemetry-javaagent.jar -Dotel.exporter.otlp.endpoint=http://1.2.3.4:4317

on start I get:

ERROR io.opentelemetry.javaagent.OpenTelemetryAgent
java.lang.IllegalStateException: agent initializer should be loaded in boot loader
        at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:39)
        at io.opentelemetry.javaagent.OpenTelemetryAgent.startAgent(OpenTelemetryAgent.java:57)
        at io.opentelemetry.javaagent.OpenTelemetryAgent.premain(OpenTelemetryAgent.java:45)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)

Steps to reproduce

Append arguments above. Start server.

Expected behavior

It starts without an error.

Actual behavior

An error. No other javaagent is present.

Javaagent or library instrumentation version

v1.3.2

Environment

JDK: jdk-11.0.15.1 OS: OEL8

Additional context

Also tested with the 2 alpha, and an old version (1.6.2)

[laurit]

@macau23 what is the weblogic version?

[macau23]

@laurit 14.1.1

[laurit]

@macau23 I tried with 14.1.1.0.0 and ran JAVA_OPTIONS="-javaagent:/path/to/opentelemetry-javaagent.jar" ./startWebLogic.sh and it did not fail. Are you doing something different to start weblogic? Does it still fail when you set USE_JVM_SYSTEM_LOADER=true It probably isn't difficult to fix, we just need a way to reproduce this issue.

[macau23]

@laurit That is a difference: I am changing the ServerStart arguments for a given WebLogic Server within the Admin Console.

[laurit]

@macau23 could you provide instructions what needs to be done to set up the server so that this issue could be reproduced.

[macau23]

This will do it for WebLogic server someServer. Restart someServer afterwards.

#!/path/to/oracle_common/common/bin/wlst.sh
connect( userConfigFile='/path/to/login.secure',
        userKeyFile='/path/to/loginkey.secure',
        url='t3://localhost:1234')

edit()
startEdit()

for serverName in cmo.getServers():
    theName = serverName.getName()

    if theName == "AdminServer":
        continue

    if theName != "someServer":
        continue

    cd('/Servers/' + theName + '/ServerStart/' + theName)
    currentArguments = cmo.getArguments()

    cmo.setArguments(
currentArguments +
' -Dotel.exporter.otlp.endpoint=http://11.22.33.44:4317 ' +
' -Dotel.resource.attributes=service.name=' + theName +
' -javaagent:/path/to/opentelemetry-javaagent.jar '
)

save()
activate()

disconnect()
exit()

[macau23]

export USE_JVM_SYSTEM_LOADER=true does seem to make things work! Is there a way without that?

[laurit]

@macau23 I wasn't able to reproduce this with a managed server. I configured the server start arguments from the admin console and started there managed server from there. Does it happen only when you start a managed server or can you reproduce it also by running JAVA_OPTIONS="-javaagent:/path/to/opentelemetry-javaagent.jar" ./startWebLogic.sh ?

[macau23]

If I have export USE_JVM_SYSTEM_LOADER=true set in my environment and restart the node manager and admin server and then the weblogic server instance (someServer) the problem goes away. If I unset USE_JVM_SYSTEM_LOADER, and repeat, the problem comes back.