Issue with dynamic injection after certain reactive classes are loaded

[grcevski]

I've been working on a new Java agent that instruments a few bootstrap classes and one Netty class. I'm setting up the agent as this, with a few options to allow for retransformation etc.

        AgentBuilder builder = new AgentBuilder.Default()
                .with(new AgentBuilder.LocationStrategy() {
                    @Override
                    public ClassFileLocator classFileLocator(ClassLoader classLoader, JavaModule module) {
                        return ClassFileLocator.ForClassLoader.of(classLoader);
                    }
                })
                .disableClassFormatChanges()
                .ignore(none())
                .with(AgentBuilder.RedefinitionStrategy.RETRANSFORMATION) 
                .with(AgentBuilder.InitializationStrategy.NoOp.INSTANCE) 
                .with(AgentBuilder.TypeStrategy.Default.REDEFINE) 
                ;

I'm later using ClassInjector injector = ClassInjector.UsingInstrumentation.of(tempDir, BOOTSTRAP, instrumentation); to inject all classes that I want into the bootstrap class loader. Full code is available here, although I haven't cleaned up things yet.

I'm opening an issue about something I encountered, which I think it might be a bytebuddy bug, but I'm not 100% sure.

For my use-case, I'm particularly interested to have this agent dynamically loaded, and it works as expected except in one specific scenario. Namely, I'm able to instrument with dynamic injection various applications, ranging from simple Java programs performing outbound HTTP calls or SpringBoot application etc. However, through a user bug report, I encountered a SpringBoot application which was built using the Netty reactor packages for the HTTP web client. This application doesn't work as expected.

Namely, this little bit of code present in the application Jar, breaks dynamic injection in a way that ByteBuddy cannot discover the classes set to be instrumented with advices, but only those:

(sample code)

package com.example.httpclient;

import io.netty.handler.ssl.util.InsecureTrustManagerFactory;
import org.springframework.http.client.reactive.ReactorClientHttpConnector;
import org.springframework.stereotype.Service;
import org.springframework.web.reactive.function.client.WebClient;
import reactor.core.publisher.Mono;
import reactor.netty.http.client.HttpClient;
import reactor.netty.http.HttpProtocol;

import java.time.Duration;

@Service
public class HttpClientService {

    private HttpClient httpClient = HttpClient.create()
            .secure(spec -> spec.sslContext(
                    io.netty.handler.ssl.SslContextBuilder.forClient()
                            .trustManager(InsecureTrustManagerFactory.INSTANCE)
            ))
            .protocol(HttpProtocol.HTTP11);
    private final WebClient webClient;

    public HttpClientService() {
        this.webClient = WebClient.builder()
                .clientConnector(new ReactorClientHttpConnector(httpClient))
                .codecs(configurer -> configurer.defaultCodecs().maxInMemorySize(1024 * 1024))
                .build();
    }

    /**
     * Makes a GET request to the specified URL
     */
    public String makeGetRequest(String url) {
        try {
            return webClient.get()
                    .uri(url)
                    .retrieve()
                    .bodyToMono(String.class)
                    .timeout(Duration.ofSeconds(10))
                    .block();
        } catch (Exception e) {
            return "Error making request: " + e.getMessage();
        }
    }
}

I don't know which one of those packages causes the issue, but I suspect is not reactor.core.publisher.Mono, because I used that in another place and had no issues.

So this is essentially what happens:

1. If I use -agentlib on the Java command line then instrumentation works, in all cases.
2. If I try to dynamically inject, no instrumentations work if that reactor.netty web client is used. When I enable the AgentBuilder.Listener.StreamWriting printer I can see that ByteBuddy doesn't find any of the classes that I want to instrument. It does find many other classes from the same class loader, but just not the ones that I have in my advices.
3. To further investigate this, I made a simple advice on the bootclass loader, saying that I want to instrument all methods in java.lang.Integer. Again when I use this with -agentlib on the command line, it works well. When I load dynamically, the print output doesn't list java.lang.Integer. It does find java.lang.Integer$IntegerCache suprisingly, and I do see something like:

[Byte Buddy] DISCOVERY java.lang.Integer$IntegerCache [null, module java.base, Thread[#68,Attach Listener,9,system], loaded=true]
[Byte Buddy] IGNORE java.lang.Integer$IntegerCache [null, module java.base, Thread[#68,Attach Listener,9,system], loaded=true]
[Byte Buddy] COMPLETE java.lang.Integer$IntegerCache [null, module java.base, Thread[#68,Attach Listener,9,system], loaded=true]

4. Then from the output I saw that ByteBuddy discovered java.lang.String, so I changed my advice to instrument java.lang.String instead, and then it found java.lang.Integer but not java.lang.String :-).

So being desperate I tried a workaround, after the agentmain calls premain, I manually called inst.getAllLoadedClasses(), looked for java.lang.Integer and I called inst.retransformClasses(clazz); on it. Then the instrumentation worked! So I have a workaround for now, but I thought I'd open an issue to see if this is some sort of a side-effect of how I've set things up or, a genuine bug in bytebuddy. It seems to me that it's related to discovery.

Since this is Netty and Java, I tried the OpenTelemetry Java Agent to see if this is perhaps an issue with how I use the AgentBuilder code or something else. They have the same problem it seems. If I instrument this test application that includes that netty reactor client, and I use -agentlib to load the Agent on boot then the instrumentation works. But, if I dynamically inject the OpenTelemetry Java Agent, their instrumentation doesn't work just like mine doesn't.

[grcevski]

Sorry I forgot to mention, I'm using version 1.17.6.

[raphw]

Possibly, this is because classes are loaded by the agent during instrumentation? If so, the classes are ignored to avoid circularities. You can try reiteration: https://github.com/raphw/byte-buddy/blob/master/byte-buddy-dep/src/main/java/net/bytebuddy/agent/builder/AgentBuilder.java#L7139

[grcevski]

Oh interesting, I'll look into it. I know one of them it definitely isn't io.netty.handler.ssl.SslHandler because I didn't put netty as dependency in my agent, but I need to check if the discovery fails for that one too. I'm still surprised though that this only happens when this reactor based client is loaded, I don't see the problem with dynamic injection skipping classes when I try it on other jars using the same code.

Thanks for the reiteration pointer, I'll look into that as well.