jetty.project icon indicating copy to clipboard operation
jetty.project copied to clipboard
verified publisher icon jetty

Intermittent IllegalStateException: "No Client ALPNProcessors!" with ProxyHandler and HTTP/2

Open mperktold opened this issue 2 months ago • 9 comments

Jetty Version 12.0.25

Jetty Environment core, e10

HTTP version HTTP 2

Java version/vendor (use: java -version) Temurin 21.0.7

Question Our clients sometimes experience the following error when trying to make a request that is proxied using ProxyHandler.

No Client ALPNProcessors!
	java.lang.IllegalStateException: No Client ALPNProcessors!
		at org.eclipse.jetty.alpn.client.ALPNClientConnectionFactory.<init>(ALPNClientConnectionFactory.java:49)
		at org.eclipse.jetty.client.transport.HttpClientTransportDynamic.newConnection(HttpClientTransportDynamic.java:259)
		at org.eclipse.jetty.io.ssl.SslClientConnectionFactory.newConnection(SslClientConnectionFactory.java:140)
		at org.eclipse.jetty.io.Transport.newConnection(Transport.java:159)
		at org.eclipse.jetty.io.ClientConnector.newConnection(ClientConnector.java:572)
		at org.eclipse.jetty.io.ClientConnector$ClientSelectorManager.newConnection(ClientConnector.java:675)
		at org.eclipse.jetty.io.ManagedSelector.createEndPoint(ManagedSelector.java:387)
		at org.eclipse.jetty.io.ManagedSelector$CreateEndPoint.run(ManagedSelector.java:1057)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
		at java.base/java.lang.Thread.run(Unknown Source)
		Suppressed: java.util.ServiceConfigurationError: 
			at org.eclipse.jetty.util.ServiceLoaderSpliterator$ServiceProvider.get(ServiceLoaderSpliterator.java:101)
			at org.eclipse.jetty.alpn.client.ALPNClientConnectionFactory.lambda$new$0(ALPNClientConnectionFactory.java:57)
			at org.eclipse.jetty.util.ServiceLoaderSpliterator.tryAdvance(ServiceLoaderSpliterator.java:46)
			at java.base/java.util.Spliterator.forEachRemaining(Unknown Source)
			at java.base/java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
			at org.eclipse.jetty.alpn.client.ALPNClientConnectionFactory.<init>(ALPNClientConnectionFactory.java:52)
			... 10 more
		Caused by: java.util.ServiceConfigurationError: org.eclipse.jetty.io.ssl.ALPNProcessor$Client: Provider org.eclipse.jetty.alpn.bouncycastle.client.BouncyCastleClientALPNProcessor not found
			at java.base/java.util.ServiceLoader.fail(Unknown Source)
			at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(Unknown Source)
			at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown Source)
			at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown Source)
			at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
			at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
			at org.eclipse.jetty.util.ServiceLoaderSpliterator.tryAdvance(ServiceLoaderSpliterator.java:37)
			... 13 more
		Suppressed: java.util.ServiceConfigurationError: 
			at org.eclipse.jetty.util.ServiceLoaderSpliterator$ServiceProvider.get(ServiceLoaderSpliterator.java:101)
			at org.eclipse.jetty.alpn.client.ALPNClientConnectionFactory.lambda$new$0(ALPNClientConnectionFactory.java:57)
			at org.eclipse.jetty.util.ServiceLoaderSpliterator.tryAdvance(ServiceLoaderSpliterator.java:46)
			at java.base/java.util.Spliterator.forEachRemaining(Unknown Source)
			at java.base/java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
			at org.eclipse.jetty.alpn.client.ALPNClientConnectionFactory.<init>(ALPNClientConnectionFactory.java:52)
			... 10 more
		Caused by: java.util.ServiceConfigurationError: org.eclipse.jetty.io.ssl.ALPNProcessor$Client: Provider org.eclipse.jetty.alpn.conscrypt.client.ConscryptClientALPNProcessor not found
			at java.base/java.util.ServiceLoader.fail(Unknown Source)
			at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(Unknown Source)
			at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown Source)
			at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown Source)
			at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
			at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
			at org.eclipse.jetty.util.ServiceLoaderSpliterator.tryAdvance(ServiceLoaderSpliterator.java:37)
			... 13 more
		Suppressed: java.util.ServiceConfigurationError: 
			at org.eclipse.jetty.util.ServiceLoaderSpliterator$ServiceProvider.get(ServiceLoaderSpliterator.java:101)
			at org.eclipse.jetty.alpn.client.ALPNClientConnectionFactory.lambda$new$0(ALPNClientConnectionFactory.java:57)
			at org.eclipse.jetty.util.ServiceLoaderSpliterator.tryAdvance(ServiceLoaderSpliterator.java:46)
			at java.base/java.util.Spliterator.forEachRemaining(Unknown Source)
			at java.base/java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
			at org.eclipse.jetty.alpn.client.ALPNClientConnectionFactory.<init>(ALPNClientConnectionFactory.java:52)
			... 10 more
		Caused by: java.util.ServiceConfigurationError: org.eclipse.jetty.io.ssl.ALPNProcessor$Client: Provider org.eclipse.jetty.alpn.java.client.JDK9ClientALPNProcessor not found
			at java.base/java.util.ServiceLoader.fail(Unknown Source)
			at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(Unknown Source)
			at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown Source)
			at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown Source)
			at java.base/java.util.ServiceLoader$2.hasNext(Unknown Source)
			at java.base/java.util.ServiceLoader$3.hasNext(Unknown Source)
			at org.eclipse.jetty.util.ServiceLoaderSpliterator.tryAdvance(ServiceLoaderSpliterator.java:37)
			... 13 more

When this error happens once, it happens on every following request.

The classes are present on the classpath, otherwise it would always fail. Therefore, I suspect ClassLoader issues. The path where this happens directly uses jetty Handlers without servlets, so i think there shouldn't be any ClassLoader magic. But there are other parts in our app which do use Servlets, so in theory a ClassLoader could leak to another thread, but I'm only guessing here.

Unfortunately, we cannot reproduce this locally so far. Do you have any ideas what could be wrong here?

mperktold avatar Nov 01 '25 07:11 mperktold

We call ServiceLoader.load(ALPNProcessor.Client.class), which uses Thread.currentThread().getContextClassLoader() (TCCL) to load the class.

What can happen is that some code changes the TCCL to a ClassLoader that cannot load the providers.

It is a strange situation because there are 3 different providers classes in the exceptions above, meaning that the ServiceLoader was able to find the META-INF/services files, and retrieve the class names from them, but then it was not able to load the classes, despite having found the files.

To verify this hypothesis, would you be able to override HttpClientTransportDynamic.newConnection() to print the TCCL and see whether there is a difference between the case where the connection can be created, and the case where is cannot be created?

Is your application changing the TCCL and forgetting to properly restore it (for example, in case of exceptions)?

sbordet avatar Nov 01 '25 09:11 sbordet

To verify this hypothesis, would you be able to override HttpClientTransportDynamic.newConnection() to print the TCCL and see whether there is a difference between the case where the connection can be created, and the case where is cannot be created?

Good idea, we'll try that, thanks.

Is your application changing the TCCL and forgetting to properly restore it (for example, in case of exceptions)?

No we're not changing the class loader ourselves.

mperktold avatar Nov 05 '25 08:11 mperktold

Apparently the class loader is not the problem. It's always a jdk.internal.loader.ClassLoaders.AppClassLoader, both when it works and when it doesn't.

Are there any other things that can prevent loading classes? Something related to ThreadLocals maybe?

mperktold avatar Nov 06 '25 11:11 mperktold

I can now reproduce this in a minimal example:

public static void main(String[] args) throws Exception {
    var server = new Server(8080);

    var webApp = new WebAppContext();
    webApp.setBaseResource(new URLResourceFactory().newResource("/"));
    webApp.addServlet(ALPNTestServlet.class, "/*");

    server.setHandler(webApp);

    server.start();
    server.join();
}

public static class ALPNTestServlet extends HttpServlet {

    @Override
    protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws IOException {
        for (var processor : ServiceLoader.load(ALPNProcessor.Client.class))
            resp.getWriter().write(processor.toString());
    }
}

This tries to load ALPNProcessor.Client providers inside a servlet, which uses a WebAppClassLoader. It fails with a similar stacktrace (I really only have JDK9ClientALPNProcessor on the classpath for this example):

java.util.ServiceConfigurationError: org.eclipse.jetty.io.ssl.ALPNProcessor$Client: Provider org.eclipse.jetty.alpn.java.client.JDK9ClientALPNProcessor not found
	at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:593)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.nextProviderClass(ServiceLoader.java:1219)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1228)
	at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1273)
	at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1309)
	at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1393)
	at jetty.ClassLoaderIssue$ALPNTestServlet.doGet(ClassLoaderIssue.java:52)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:527)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614)
	at org.eclipse.jetty.ee10.servlet.ServletHolder.handle(ServletHolder.java:752)
	at org.eclipse.jetty.ee10.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1620)
	at org.eclipse.jetty.ee10.servlet.ServletHandler$MappedServlet.handle(ServletHandler.java:1554)
	at org.eclipse.jetty.ee10.servlet.ServletChannel.dispatch(ServletChannel.java:807)
	at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:442)
	at org.eclipse.jetty.ee10.servlet.ServletHandler.handle(ServletHandler.java:469)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:575)
	at org.eclipse.jetty.ee10.servlet.SessionHandler.handle(SessionHandler.java:719)
	at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1220)
	at org.eclipse.jetty.server.Server.handle(Server.java:195)
	at org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:680)
	at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:411)
	at org.eclipse.jetty.server.internal.HttpConnection$FillableCallback.succeeded(HttpConnection.java:1809)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:54)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:1009)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1239)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1194)
	at java.base/java.lang.Thread.run(Thread.java:1583)

I've tried to fix this by excluding the corresponding packages from the hidden classes in the following way, which didn't help:

webApp.getHiddenClassMatcher().exclude("org.eclipse.jetty.client.", "org.eclipse.jetty.alpn.client.");

What does help is to exclude the whole JAR file that contains the implementation. To find the JAR file, I wrote this helper method:

private static Optional<String> getALPNProcessorClientJarLocation() {
    // Loading the implementation using the main Thread's ClassLoader to find the JAR that contains it,
    // so we can exclude that from hidden classes
    return ServiceLoader.load(ALPNProcessor.Client.class).findFirst()
        .map(Object::getClass)
        .map(cls -> cls.getResource(cls.getSimpleName() + ".class"))
        .map(URL::toString)
        .filter(res -> res.startsWith("jar:") && res.contains("!"))
        .map(res -> res.substring("jar:".length(), res.lastIndexOf('!')));
}

And used it inside main like this:

getALPNProcessorClientJarLocation().ifPresent(webApp.getHiddenClassMatcher()::exclude);

This is quite cumbersome to do. Does this work as intended?

Back to the ProxyHandler issue, I guess it really was the ClassLoader and we logged it at the wrong point in time. It's still odd that the ClassLoader would change, because the way it is set and restored in ContextHandler looks pretty solid. I guess the only chance to leak a ClassLoader would be in async scenarios, but I don't really understand how that is currently managed.

mperktold avatar Nov 07 '25 11:11 mperktold

Check out the testcase https://github.com/jetty/jetty.project/blob/fix/12.1.x/4652/classmatcher-protect-getresource/jetty-ee11/jetty-ee11-tests/jetty-ee11-test-integration/src/test/java/org/eclipse/jetty/ee11/test/ClassLoaderProtectResourcesTest.java

In particular the method protectServerResource

Your usage would look like ...

ClassLoader serverClassLoader = server.getClass().getClassLoader();
String resourceName = "META-INF/services/org.eclipse.jetty.io.ssl.ALPNProcessor$Client";
protectServerResource(serverClassLoader, resourceName, webApp);

Even if you don't use this, you might learn something about features of our URIUtil class which will simplify your implementation.

joakime avatar Nov 07 '25 14:11 joakime

That's helpful, thanks!

So using this as is, I don't get any exceptions, I just get no providers. Changing protectServerResource to do the opposite, namely to exclude the JAR from hidden classes instead of adding it, works for me, which makes sense because it results in exactly the same call to ClassMatcher.exclude.

I guess I could use these approaches as a workaround, but what would be the best practice here?

And is it intended that by default, you can load resources from Jetty JARs, but not classes? At least @sbordet seems to be quite surprised about that himself:

It is a strange situation because there are 3 different providers classes in the exceptions above, meaning that the ServiceLoader was able to find the META-INF/services files, and retrieve the class names from them, but then it was not able to load the classes, despite having found the files.

mperktold avatar Nov 07 '25 15:11 mperktold

Oh, and why doesn't excluding the packages from the ClassMatcher help? From my understanding, that should be the only missing part, so the ClassLoader should see both, the resource in META-INF and the implementing class.

mperktold avatar Nov 07 '25 16:11 mperktold

Oh, and why doesn't excluding the packages from the ClassMatcher help?

If you use .exclude("org.eclipse.jetty.alpn.client.") then only request to load classes, from the ClassLoader are filtered. But if you use .exclude("file://path/to/jetty-home/lib/alpn/jetty-alpn-client-12.1.4.jar") then you are filtering all content, even ClassLoader.getResource() calls (which is what the ServiceLoader uses).

This is an old topic ...

  • #4652

joakime avatar Nov 07 '25 16:11 joakime

I see.

Going back a step, our use case is one JAR plus libraries with an embedded Jetty server, no WAR files. We depend Jetty core libs, both client and server stuff, as well as ee10.

In such a scenario, should we disable hiding classes altogether? What is the best way to do that? Overriding WebAppContext.configureClassLoader? Or directly using ServletContextHandler instead of WebAppContext?

mperktold avatar Nov 08 '25 07:11 mperktold