Unable to load SPI in springboot

[crossoverJie]

Describe the bug

When using java.net.spi.InetAddressResolverProvider, the custom SPI does not load properly.

Steps to reproduce

META-INF/services/java.net.spi.InetAddressResolverProvider: com.example.demo.MyAddressResolverProvider

java  -javaagent:opentelemetry-javaagent.jar \
      -jar target/demo-0.0.1-SNAPSHOT.jar

Expected behavior

Customized implementation takes effect successfully.

Actual behavior

Custom implementation does not take effect.

Javaagent or library instrumentation version

1.32.0

Environment

JDK:21 OS: Linux/MacOS

Additional context

When I use this command (without a javaagent):

java -jar  target/demo-0.0.1-SNAPSHOT.jar

It's work fine.

And the classloader is correct: URLClassPath$Loader@1211

When I use this command (with javaagent):

java  -javaagent:opentelemetry-javaagent.jar \
      -jar target/demo-0.0.1-SNAPSHOT.jar

The classloader is incorrect: URLClassPath$JarLoader@814. The JarLoader cannot load the springboot jar; it can only load the normal Maven project jar.

And this method cannot be called org.springframework.boot.loader.launch.JarLauncher#main.

If I don't use Spring Boot instead of a normal Maven project, it'll also work fine.

[laurit]

Do I understand correctly that the issue is that because agent tries to establish network connections it will trigger loading java.net.spi.InetAddressResolverProvider before spring boot has set up its class loader and because of that your custom implementation of java.net.spi.InetAddressResolverProvider won't be used? If that is the case have you considered packaging your java.net.spi.InetAddressResolverProvider implementation and the the corresponding META-INF/services file in the root of the jar along with spring boot launcher code. I think this way your provider should be found even if the agent triggers initializing the networking code.

[crossoverJie]

If that is the case have you considered packaging your java.net.spi.InetAddressResolverProvider implementation and the the corresponding META-INF/services file in the root of the jar along with spring boot launcher code.

in the root of the jar

Thank you for your reply.

Could you please explain in detail how to achieve this?

[laurit]

Could you please explain in detail how to achieve this?

No, I can't, I'm not a spring boot user and don't know exactly how you'd do this. Spring boot questions are best asked from spring boot channels or stack overflow. In your place I'd start by manually modifying the jar and seeing whether it resolves the issue. If it helps I'd check the documentation for spring boot plugin for the build system in use, for example https://docs.spring.io/spring-boot/docs/current/maven-plugin/reference/htmlsingle/#packaging.layers.configuration looks promising.

[serkan-ozal]

Hi @laurit @crossoverJie,

As @laurit mentioned, system-wide InetAddressResolver in the InetAddress is initialized by the OTEL Java agent through system classloader without your custom InetAddressResolver. Because your custom InetAddressResolver is not visible to system classloader, but Spring Boot's custom classloader.

When I have debug the code, I have seen that InetAddress initialization is triggered by HostResourceProvider (which calls HostResource.get() and then InetAddress.getLocalHost().getHostName()).

As a workaround, you can disable HostResourceProvider

• by environment variable:

OTEL_JAVA_DISABLED_RESOURCE_PROVIDERS=otel.java.disabled.resource.providers.HostResourceProvider

• or by system property:

-Dotel.java.disabled.resource-providers=otel.java.disabled.resource.providers.HostResourceProvider

[crossoverJie]

@serkan-ozal Thank you for your reply.

-Dotel.java.disabled.resource-providers=otel.java.disabled.resource.providers.HostResourceProvider

I have added this system property, but it's not work for me.

I have created a demo repo; if you have time, you can give it a try.

[serkan-ozal]

@crossoverJie Sorry, it is my bad. I made an error while copy-pasting full classname of the HostResourceProvider :)

For the correct workaround, you can disable HostResourceProvider

• by environment variable:

OTEL_JAVA_DISABLED_RESOURCE_PROVIDERS=io.opentelemetry.instrumentation.resources.HostResourceProvider

• or by system property:

-Dotel.java.disabled.resource-providers=io.opentelemetry.instrumentation.resources.HostResourceProvider

I have tried with your example and verified that it works.

[crossoverJie]

@serkan-ozal Thank you for your help. It works for me.

I found HostResourceProvider provides the attributes: host.name and host.arch, I will miss these attributes if I disable it.

Is there any other better solution?

[Cirilla-zmh]

Is there any other better solution?

Seems like the loading of InetAddressResolver leads this case. Can you load your MyAddressResolverProvider after initialization of agent, and replace the global resolver by reflection?

[serkan-ozal]

Hi @laurit

I think the easiest fix for this issue might be resetting (setting to null) resolver field of the InetAddress class by reflection just after we get the host name in the HostResource. So when it is called by the user application later, InetAddressResolver will be resolved again through the thread context classloader (or system classloader if not set) and will be able to detect user defined InetAddressResolver.

I have tried this solution and verified that it works. However, since we use reflection to set the private resolver field of the InetAddress class, we need to open java.base/java.net module to our agent's module for Java 9+. To do that, I have defined JavaNetInitializer class in the javaagent-tooling-java9 module and JavaNetInitializer opens java.base/java.net to our agent's module by ClassInjector.UsingInstrumentation.redefineModule:

    JavaModule currentModule = JavaModule.ofType(JavaNetInitializer.class);
    JavaModule javaBase = JavaModule.ofType(ClassLoader.class);
    if (javaBase != null && javaBase.isNamed() && currentModule != null) {
      ClassInjector.UsingInstrumentation.redefineModule(
          instrumentation,
          javaBase,
          Collections.emptySet(),
          Collections.emptyMap(),
          Collections.singletonMap("java.net", Collections.singleton(currentModule)),
          Collections.emptySet(),
          Collections.emptyMap());
    }

So, if that makes sense for you, I can send a PR for further discussions.