newrelic-java-agent icon indicating copy to clipboard operation
newrelic-java-agent copied to clipboard
verified publisher icon newrelic

Add support for WSO2 Identity Server

Open jasonjkeller opened this issue 2 years ago • 1 comments

Problem Statement

There is an incompatibility between the New Relic Java agent and WSO2 Identity Server that causes the server to crash on startup with the following java.lang.VerifyError: Bad type on operand stack:

[2023-07-26 11:12:23,651] [] ERROR {org.wso2.carbon.tomcat.internal.ServerManager} - tomcat life-cycle exception org.apache.catalina.LifecycleException: Failed to initialize component [CarbonTomcatRealm[StandardEngine[Catalina].StandardHost[localhost].StandardContext[/]]]
	at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
	at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:139)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:173)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:912)
	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:263)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.wso2.carbon.tomcat.ext.service.ExtendedStandardService.startInternal(ExtendedStandardService.java:52)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:927)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
	at org.wso2.carbon.tomcat.internal.CarbonTomcat.start(CarbonTomcat.java:113)
	at org.wso2.carbon.tomcat.internal.ServerManager$1.run(ServerManager.java:167)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    org/apache/catalina/connector/Request.getUserPrincipal()Ljava/security/Principal; @69: invokeinterface
  Reason:
    Type 'java/lang/Object' (current frame, stack[2]) is not assignable to 'java/lang/Throwable'
  Current Frame:
    bci: @69
    flags: { }
    locals: { 'org/apache/catalina/connector/Request', 'org/ietf/jgss/GSSCredential', integer, 'java/lang/Object' }
    stack: { 'org/apache/juli/logging/Log', 'java/lang/String', 'java/lang/Object' }
  Bytecode:
    0000000: 2ab4 00cc c106 6799 005e 2ab4 00cc c006
    0000010: 67b9 066b 0100 4c2b c600 4002 3d2b b906
    0000020: 7001 003d a700 264e b201 9bb2 019d 1306
    0000030: 7204 bd00 0559 032a b400 ccb9 0616 0100
    0000040: 53b6 0328 2db9 01aa 0300 1c9d 000d 2ab6
    0000050: 0675 a700 044e 01b0 2ab4 00cc c006 67b9
    0000060: 0677 0100 b02a b400 ccb0
  Exception Handler Table:
    bci [29, 36] => handler: 39
    bci [29, 36] => handler: 39
    bci [78, 82] => handler: 85
  Stackmap Table:
    full_frame(@39,{Object[#3],Object[#1645],Integer},{Object[#5]})
    same_frame(@74)
    same_locals_1_stack_item_frame(@85,Object[#1337])
    same_frame(@86)
    chop_frame(@88,1)
    chop_frame(@101,1)

	at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
	at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3373)
	at java.base/java.lang.Class.getConstructor0(Class.java:3578)
	at java.base/java.lang.Class.getConstructor(Class.java:2271)
	at org.apache.tomcat.util.modeler.modules.MbeansDescriptorsIntrospectionSource.isBeanCompatible(MbeansDescriptorsIntrospectionSource.java:169)
	at org.apache.tomcat.util.modeler.modules.MbeansDescriptorsIntrospectionSource.supportedType(MbeansDescriptorsIntrospectionSource.java:142)
	at org.apache.tomcat.util.modeler.modules.MbeansDescriptorsIntrospectionSource.initMethods(MbeansDescriptorsIntrospectionSource.java:264)
	at org.apache.tomcat.util.modeler.modules.MbeansDescriptorsIntrospectionSource.createManagedBean(MbeansDescriptorsIntrospectionSource.java:307)
	at org.apache.tomcat.util.modeler.modules.MbeansDescriptorsIntrospectionSource.execute(MbeansDescriptorsIntrospectionSource.java:79)
	at org.apache.tomcat.util.modeler.modules.MbeansDescriptorsIntrospectionSource.loadDescriptors(MbeansDescriptorsIntrospectionSource.java:70)
	at org.apache.tomcat.util.modeler.Registry.load(Registry.java:610)
	at org.apache.tomcat.util.modeler.Registry.findManagedBean(Registry.java:516)
	at org.apache.tomcat.util.modeler.Registry.registerComponent(Registry.java:639)
	at org.apache.catalina.util.LifecycleMBeanBase.register(LifecycleMBeanBase.java:156)
	at org.apache.catalina.util.LifecycleMBeanBase.initInternal(LifecycleMBeanBase.java:57)
	at org.apache.catalina.realm.RealmBase.initInternal(RealmBase.java:1103)
	at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
	... 11 more

The following files for the weaved org_apache_catalina_connector_Request class can be downloaded here:

org_apache_catalina_connector_Request5354683291284891068.old
org_apache_catalina_connector_Request8672116866306130258.new
org_apache_catalina_connector_Request4994425864965866099.new.class

In the org_apache_catalina_connector_Request5354683291284891068.old (original) bytecode we see:

   L2
    LINENUMBER 2669 L2
   FRAME FULL [org/apache/catalina/connector/Request org/ietf/jgss/GSSCredential I] [java/lang/Exception]
    ASTORE 3

In the org_apache_catalina_connector_Request8672116866306130258.new (weaved) bytecode we see:

   L2
    LINENUMBER 2669 L2
   FRAME FULL [org/apache/catalina/connector/Request org/ietf/jgss/GSSCredential I] [java/lang/Object]
    ASTORE 3

This shows that after weaving org/apache/catalina/connector/Request the java/lang/Exception object is changed to java/lang/Object which matches the error message Reason: Type 'java/lang/Object' (current frame, stack[2]) is not assignable to 'java/lang/Throwable'

Repro

To repro:

  • Download WSO2 Identity Server ZIP here: wso2is-6.1.0.zip
  • Unzip the archive and navigate to /wso-repro/wso2is-6.1.0/bin
  • Add -javaagent:/path/to/newrelic/newrelic.jar to the /wso-repro/wso2is-6.1.0/bin/wso2server.sh server startup script (see screenshot). Optionally, add -Dnewrelic.debug=true to log the name/location of weaved agent classes.
    • javaagent
  • Using Java 11 or 17, start the WSO2 Identity Server with the startup script: sh wso2server.sh

Workarounds

Disabling the Java agent's tomcat and servlet instrumentation resolves the issue, though it will result in the loss of transactions that would normally be started at the servlet level. To disable this instrumentation you can add the following to the pre-existing class_transformer stanza of the agent's yaml config file:

  class_transformer:
    enabled: true

    com.newrelic.instrumentation.tomcat-8.5.2:
      enabled: false

    com.newrelic.instrumentation.servlet-2.4:
      enabled: false

The same can be set via system properties:

-Dnewrelic.config.class_transformer.com.newrelic.instrumentation.tomcat-8.5.2.enabled=false
-Dnewrelic.config.class_transformer.com.newrelic.instrumentation.servlet-2.4.enabled=false

Here are the other versions of the Java agent's tomcat and servlet instrumentation, which can be disabled in the same way if the issue occurs with them:

com.newrelic.instrumentation.servlet-5.0
com.newrelic.instrumentation.servlet-6.0

com.newrelic.instrumentation.tomcat-7
com.newrelic.instrumentation.tomcat-7.0.70
com.newrelic.instrumentation.tomcat-8.5.0
com.newrelic.instrumentation.tomcat-10

To attempt to add visibility for the gaps caused by disabling instrumentation you can use one of the available custom instrumentation options.

The easiest approach is to run a thread profile and add custom instrumentation via the UI editor.

jasonjkeller avatar Jul 26 '23 18:07 jasonjkeller

https://new-relic.atlassian.net/browse/NR-146164

workato-integration[bot] avatar Jul 26 '23 18:07 workato-integration[bot]

This issue won't be actioned.

workato-integration[bot] avatar Dec 04 '24 00:12 workato-integration[bot]