Crash in OpenJDK 17.0.4

[jlintz]

It appears that the Datadog instrumentation agent is attempting to rewrite java.management's NotificationBroadcasterSupport$SendNotifJob to implement Datadog's datadog.trace.bootstrap.FieldBackedContextAccessor interface. Since java.management is a named JDK module, it does not permit reading arbitrary (in this case, an unnamed) modules. This means that SendNotifJob cannot allow of a sudden implement an interface coming from such arbitrary module. We believe that's why the JVM throws an IllegalAccessError. This is further corroborated by the fact that adding --add-reads java.management=ALL-UNNAMED to the java invocation "fixes" the issue. We're not sure what the appropriate fix within Datadog agent is, but at a minimum, perhaps the need to add --add-reads needs to be called out in documentation.

This issue does NOT occur in 17.0.3

Stack Trace:

Caused by: java.lang.IllegalAccessError: superinterface check failed: class javax.management.NotificationBroadcasterSupport$SendNotifJob (in module java.management) cannot access class datadog.trace.bootstrap.FieldBackedContextAccessor (in unnamed module @0x2f2bf0e2) because module java.management does not read unnamed module @0x2f2bf0e2
                at java.management/javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:248)
                at java.management/javax.management.MBeanServerDelegate.sendNotification(MBeanServerDelegate.java:211)
                at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.sendNotification(DefaultMBeanServerInterceptor.java:1478)
                at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1877)
                at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:960)
                at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:895)
                at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:320)
                at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:523)
                at com.mchange.v2.c3p0.management.DynamicPooledDataSourceManagerMBean.reinitialize(DynamicPooledDataSourceManagerMBean.java:252)
                at com.mchange.v2.c3p0.management.DynamicPooledDataSourceManagerMBean.<init>(DynamicPooledDataSourceManagerMBean.java:173)
                at com.mchange.v2.c3p0.management.ActiveManagementCoordinator.attemptManagePooledDataSource(ActiveManagementCoordinator.java:158)
                at com.mchange.v2.c3p0.C3P0Registry.incorporate(C3P0Registry.java:277)
                at com.mchange.v2.c3p0.C3P0Registry.reregister(C3P0Registry.java:314)
                at com.mchange.v2.c3p0.impl.PoolBackedDataSourceBase.<init>(PoolBackedDataSourceBase.java:289)
                at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.<init>(AbstractPoolBackedDataSource.java:74)
                at com.mchange.v2.c3p0.PoolBackedDataSource.<init>(PoolBackedDataSource.java:43)
                at com.mchange.v2.c3p0.PoolBackedDataSource.<init>(PoolBackedDataSource.java:46)
                at com.mchange.v2.c3p0.PoolBackedDataSource.<init>(PoolBackedDataSource.java:50)
                at com.mchange.v2.c3p0.DataSources.pooledDataSource(DataSources.java:280)
                at com.mchange.v2.c3p0.DataSources.pooledDataSource(DataSources.java:262)
                at org.hibernate.c3p0.internal.C3P0ConnectionProvider.configure(C3P0ConnectionProvider.java:192)
                at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:107)
                at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:246)
                at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214)
                at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.buildJdbcConnectionAccess(JdbcEnvironmentInitiator.java:145)
                at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:66)
                at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:35)
                at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:101)
                at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:263)
                at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:237)
                at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214)
                at org.hibernate.id.factory.internal.DefaultIdentifierGeneratorFactory.injectServices(DefaultIdentifierGeneratorFactory.java:152)
                at org.hibernate.service.internal.AbstractServiceRegistryImpl.injectDependencies(AbstractServiceRegistryImpl.java:286)
                at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:243)
                at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214)
                at org.hibernate.boot.internal.InFlightMetadataCollectorImpl.<init>(InFlightMetadataCollectorImpl.java:176)
                at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:127)
                at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.build(MetadataBuildingProcess.java:86)
                at org.hibernate.boot.internal.MetadataBuilderImpl.build(MetadataBuilderImpl.java:479)
                at org.hibernate.boot.internal.MetadataBuilderImpl.build(MetadataBuilderImpl.java:85)
                at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:689)
<snip>

Java: OpenJDK 17.0.4 APM Version: dd-java-agent-0.102.0.jar, dd-trace-ot-0.91.0.jar, dd-trace-api-0.102.0.jar

[mcculls]

Thanks for the report @jlintz - the tracer is attempting to inject a field and accessor for tracing purposes, in this case for async context tracking. The FieldBackedContextAccessor interface should have been automatically added to the boot-classpath by tracer, and the appropriate modules updated for visibility, but it looks like one of those steps is failing on 17.0.4

While we investigate further I'll go over our global ignores list, as while there are JDK classes that require async context tracking (to follow work under the appropriate trace) this particular class is not of interest to the tracer.

So another workaround would be to exclude anything under the javax.management package from being traced:

-Ddd.trace.classes.exclude=javax.management.*

[natterd]

Hi- Following up on this, the issue seems to be fixed for javax.management.*. but we are also seeing the same issue for javax.swing. Could this be addressed also? Thank you.

[mcculls]

You can add further classes to the exclude list, for example:

-Ddd.trace.classes.exclude=javax.management.*,javax.swing.*

we'll also take that package into account going forwards

[natterd]

Thank you

[mcculls]

FYI, the change to exclude those javax packages by default was merged in #4422 and will be available in the 1.3.0 release.

[natterd]

Great, Thank you

From: Stuart McCulloch @.> Sent: Sunday, December 18, 2022 10:15 AM To: DataDog/dd-trace-java @.> Cc: David Natter @.>; Comment @.> Subject: Re: [DataDog/dd-trace-java] Crash in OpenJDK 17.0.4 (Issue #3820)

FYI, the change to exclude those javax packages by default was merged in #4422https://github.com/DataDog/dd-trace-java/pull/4422 and will be available in the 1.3.0 release.

— Reply to this email directly, view it on GitHubhttps://github.com/DataDog/dd-trace-java/issues/3820#issuecomment-1356819939, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A4LN6TCENCN47OVY5PT6YB3WN4S7DANCNFSM6AAAAAAQIWD7SQ. You are receiving this because you commented.Message ID: @.@.>>

[github-actions[bot]]

:robot: This issue has been addressed in the latest release. See full details in the Release Notes.