jetty.project icon indicating copy to clipboard operation
jetty.project copied to clipboard

Published 20 hours ago • verified publisher icon jetty

NPE in error handling leading to 100% CPU

Open konstantin-mikheev opened this issue 1 year ago • 5 comments

Jetty version(s) 12.0.7

Jetty Environment ee10

Java version/vendor (use: java -version) OpenJDK Runtime Environment Corretto-21.0.0.35.1 (build 21+35-LTS)

OS type/version Rocky Linux release 8.8 (Green Obsidian)

Description Same as #11431 but Spring Boot 3.2.4 and Jetty 12.0.7

It is a reopening of the bug #11431. The commit have not fixed the issue but added logging therefore we have more logs.

There were circular references in logs so I am attaching the full stacktrace of the NPE in error handling as a text file. The later the log, the longer is the stacktrace.

stacktrace.txt

Overall here is the short version of the stacktrace

java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "f" is null
	at org.eclipse.jetty.http.HttpFields$Mutable.remove(HttpFields.java:1418) ~[jetty-http-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.ee10.servlet.ServletContextResponse.resetContent(ServletContextResponse.java:385) ~[jetty-ee10-servlet-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:512) ~[jetty-ee10-servlet-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1206) ~[jetty-server-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1199) ~[jetty-server-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.runInContext(ServletChannelState.java:1257) ~[jetty-ee10-servlet-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.asyncError(ServletChannelState.java:818) ~[jetty-ee10-servlet-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.accept(ContextHandler.java:1187) ~[jetty-server-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.server.handler.ContextRequest.lambda$addFailureListener$1(ContextRequest.java:53) ~[jetty-server-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.server.internal.HttpChannelState.lambda$onFailure$2(HttpChannelState.java:445) ~[jetty-server-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:191) ~[jetty-util-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.http2.server.internal.HttpStreamOverHTTP2.lambda$onFailure$2(HttpStreamOverHTTP2.java:590) ~[jetty-http2-server-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:971) ~[jetty-util-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1201) ~[jetty-util-12.0.7.jar:12.0.7]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1156) ~[jetty-util-12.0.7.jar:12.0.7]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
	Suppressed: java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "f" is null
		... 16 common frames omitted

then there is multiple recurring

	Suppressed: java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "f" is null
		... 16 common frames omitted

then

Suppressed: org.eclipse.jetty.io.EofException: Reset cancel_stream_error
	at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onReset(HTTP2ServerConnectionFactory.java:158)
	at org.eclipse.jetty.http2.HTTP2Stream.notifyReset(HTTP2Stream.java:876)
	at org.eclipse.jetty.http2.HTTP2Stream.onReset(HTTP2Stream.java:586)
	at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:357)
	at org.eclipse.jetty.http2.HTTP2Session.onReset(HTTP2Session.java:346)
	at org.eclipse.jetty.http2.HTTP2Connection.onReset(HTTP2Connection.java:259)
	at org.eclipse.jetty.http2.parser.BodyParser.notifyReset(BodyParser.java:139)
	at org.eclipse.jetty.http2.parser.ResetBodyParser.onReset(ResetBodyParser.java:94)
	at org.eclipse.jetty.http2.parser.ResetBodyParser.parse(ResetBodyParser.java:61)
	at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:229)
	at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:156)
	at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:121)
	at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:349)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:512)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:410)
	... 4 common frames omitted

then a lot of

Suppressed: [CIRCULAR REFERENCE: java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "f" is null]

How to reproduce? Same as #11431

We do not have the thread dump unfortunately but we have a heap dump this time.

We could not share heapdump for security reasons but if you could point out to specific things I could look it up

konstantin-mikheev avatar Apr 08 '24 09:04 konstantin-mikheev

@konstantin-mikheev We're currently re-investigating this problem, and it seems there are at least a couple of issues at play; I managed to reproduce one of them and I'm working on a fix, but I still need to investigate the other one(s) to better understand what's going on.

I'll keep you posted shortly with my progress.

lorban avatar Apr 08 '24 16:04 lorban

@konstantin-mikheev I've created a PR containing a simple change that should mitigate this problem, it's over here: https://github.com/jetty/jetty.project/pull/11637

For some reason I'm still investigating, the HTTP2 headers can become corrupted when a H2 reset frame is sent by the client with the wrong timing, causing the servlet handling code to trap itself in an infinite loop trying to constantly run the same piece of code against the corrupted headers, then handling the exception by adding it as a suppressed to the original one then retying.

How the H2 headers corruption happens still need more investigation, even if the stack traces you managed to collect seems to lead to a client sending a H2 reset frame with a bad timing.

But a least, with the linked patch, at least Jetty will break the infinite loop. I'd be glad if you could build it, give it a try and report the outcome. Thanks!

lorban avatar Apr 09 '24 12:04 lorban

Hey! Sorry for delay. This branch of yours is based also on the jetty version with the spring integration changes there so it is not compatible with the spring version we use

But we have cherry picked the commit changes and made a jetty version with those changes exclusively based on 12.0.8 release to be sure nothing else will intervene

I have deployed the version with this patch on prod today but we will need some time for the issue to manifest itself. I will update if and when there will be more info

konstantin-mikheev avatar Apr 23 '24 11:04 konstantin-mikheev

We got a bunch of those errors over the last couple of weeks with the @lorban changes applied over 12.0.8 release and those errors have not caused 100% CPU utilisation. It was reproduced for 30+ times on a few different DCs

Here is a sample stacktrace from one of those errors:

java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "field" is null
	at org.eclipse.jetty.server.internal.HttpChannelState$ErrorResponse.getResponseHttpFields(HttpChannelState.java:1623) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.server.internal.HttpChannelState$ChannelResponse.<init>(HttpChannelState.java:1088) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.server.internal.HttpChannelState$ErrorResponse.<init>(HttpChannelState.java:1610) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.server.internal.HttpChannelState$ChannelCallback.failed(HttpChannelState.java:1549) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.ee10.servlet.ServletChannel.onCompleted(ServletChannel.java:762) ~[jetty-ee10-servlet-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:421) ~[jetty-ee10-servlet-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1292) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1285) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.runInContext(ServletChannelState.java:1257) ~[jetty-ee10-servlet-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.asyncError(ServletChannelState.java:818) ~[jetty-ee10-servlet-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.accept(ContextHandler.java:1273) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.server.handler.ContextRequest.lambda$addFailureListener$1(ContextRequest.java:53) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.server.internal.HttpChannelState.lambda$onFailure$2(HttpChannelState.java:445) ~[jetty-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:191) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.http2.server.internal.HttpStreamOverHTTP2.lambda$onFailure$1(HttpStreamOverHTTP2.java:594) ~[jetty-http2-server-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:211) ~[jetty-http2-common-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:158) ~[jetty-http2-common-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:449) ~[jetty-http2-common-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99) ~[jetty-io-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.io.ssl.SslConnection$SslEndPoint.onFillable(SslConnection.java:574) ~[jetty-io-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:390) ~[jetty-io-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:150) ~[jetty-io-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99) ~[jetty-io-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164) ~[jetty-util-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
	Suppressed: java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "f" is null
		at org.eclipse.jetty.http.HttpFields$Mutable.remove(HttpFields.java:1418) ~[jetty-http-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
		at org.eclipse.jetty.ee10.servlet.ServletContextResponse.resetContent(ServletContextResponse.java:385) ~[jetty-ee10-servlet-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
		at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:512) ~[jetty-ee10-servlet-12.0.9-gh11631-SNAPSHOT.jar:12.0.9-gh11631-SNAPSHOT]
		... 31 common frames omitted
		Suppressed: org.eclipse.jetty.io.EofException: Reset cancel_stream_error
			at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onReset(HTTP2ServerConnectionFactory.java:158)
			at org.eclipse.jetty.http2.HTTP2Stream.notifyReset(HTTP2Stream.java:876)
			at org.eclipse.jetty.http2.HTTP2Stream.onReset(HTTP2Stream.java:586)
			at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:357)
			at org.eclipse.jetty.http2.HTTP2Session.onReset(HTTP2Session.java:346)
			at org.eclipse.jetty.http2.HTTP2Connection.onReset(HTTP2Connection.java:259)
			at org.eclipse.jetty.http2.parser.BodyParser.notifyReset(BodyParser.java:139)
			at org.eclipse.jetty.http2.parser.ResetBodyParser.onReset(ResetBodyParser.java:94)
			at org.eclipse.jetty.http2.parser.ResetBodyParser.parse(ResetBodyParser.java:61)
			at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:229)
			at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:156)
			at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:121)
			at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:349)
			at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:512)
			at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)
			... 19 common frames omitted
			Suppressed: java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "f" is null
				at org.eclipse.jetty.http.HttpFields$Mutable.remove(HttpFields.java:1418)
				at org.eclipse.jetty.ee10.servlet.ServletContextResponse.resetContent(ServletContextResponse.java:385)
				at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:453)
				at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1292)
				at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1285)
				at org.eclipse.jetty.ee10.servlet.ServletChannelState.runInContext(ServletChannelState.java:1257)
				at org.eclipse.jetty.ee10.servlet.ServletChannelState.asyncError(ServletChannelState.java:818)
				at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.accept(ContextHandler.java:1273)
				at org.eclipse.jetty.server.handler.ContextRequest.lambda$addFailureListener$1(ContextRequest.java:53)
				at org.eclipse.jetty.server.internal.HttpChannelState.lambda$onFailure$2(HttpChannelState.java:445)
				at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:191)
				at org.eclipse.jetty.http2.server.internal.HttpStreamOverHTTP2.lambda$onFailure$1(HttpStreamOverHTTP2.java:594)
				at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
				at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
				at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
				... 19 common frames omitted

konstantin-mikheev avatar May 06 '24 11:05 konstantin-mikheev

@konstantin-mikheev can you try the staged 12.0.9 release? it contains many more patches than what you are doing. See the most recent comments in the release issue for staging URL -

  • https://github.com/jetty/jetty.project/issues/11702

joakime avatar May 06 '24 13:05 joakime

Hello, one question regarding this part:

For some reason I'm still investigating, the HTTP2 headers can become corrupted when a H2 reset frame is sent by the client with the wrong timing, causing the servlet handling code to trap itself in an infinite loop trying to constantly run the same piece of code against the corrupted headers, then handling the exception by adding it as a suppressed to the original one then retying.

Doesn't this make this issue a denial of service vulnerability which I expect would be of higher priority than a bug and more importantly - more people should be aware of it?

lpandzic avatar May 09 '24 10:05 lpandzic

This issue is considered complete, and was handled by

  • #11687
  • #11688

And the fix is available in Jetty 12.0.9

joakime avatar May 10 '24 10:05 joakime

Tested on 12.0.9, there are no loop and here is exception stack trace

java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "field" is null
	at org.eclipse.jetty.server.internal.HttpChannelState$ErrorResponse.getResponseHttpFields(HttpChannelState.java:1636) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.server.internal.HttpChannelState$ChannelResponse.<init>(HttpChannelState.java:1104) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.server.internal.HttpChannelState$ErrorResponse.<init>(HttpChannelState.java:1623) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.server.internal.HttpChannelState$ChannelCallback.failed(HttpChannelState.java:1562) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.ee10.servlet.ServletChannel.onCompleted(ServletChannel.java:767) ~[jetty-ee10-servlet-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:426) ~[jetty-ee10-servlet-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1292) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1285) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.runInContext(ServletChannelState.java:1289) ~[jetty-ee10-servlet-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.ee10.servlet.ServletChannelState.asyncError(ServletChannelState.java:858) ~[jetty-ee10-servlet-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.accept(ContextHandler.java:1273) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.server.handler.ContextRequest.lambda$addFailureListener$1(ContextRequest.java:53) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.server.internal.HttpChannelState.lambda$onFailure$2(HttpChannelState.java:446) ~[jetty-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:191) ~[jetty-util-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.http2.server.internal.HttpStreamOverHTTP2.lambda$onFailure$1(HttpStreamOverHTTP2.java:594) ~[jetty-http2-server-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478) ~[jetty-util-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441) ~[jetty-util-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293) ~[jetty-util-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:201) ~[jetty-util-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:311) ~[jetty-util-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979) ~[jetty-util-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209) ~[jetty-util-12.0.9.jar:12.0.9]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164) ~[jetty-util-12.0.9.jar:12.0.9]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
	Suppressed: java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "f" is null
		at org.eclipse.jetty.http.HttpFields$Mutable.remove(HttpFields.java:1437) ~[jetty-http-12.0.9.jar:12.0.9]
		at org.eclipse.jetty.ee10.servlet.ServletContextResponse.resetContent(ServletContextResponse.java:385) ~[jetty-ee10-servlet-12.0.9.jar:12.0.9]
		at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:517) ~[jetty-ee10-servlet-12.0.9.jar:12.0.9]
		... 18 common frames omitted
		Suppressed: org.eclipse.jetty.io.EofException: Reset cancel_stream_error
			at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onReset(HTTP2ServerConnectionFactory.java:158)
			at org.eclipse.jetty.http2.HTTP2Stream.notifyReset(HTTP2Stream.java:876)
			at org.eclipse.jetty.http2.HTTP2Stream.onReset(HTTP2Stream.java:586)
			at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:357)
			at org.eclipse.jetty.http2.HTTP2Session.onReset(HTTP2Session.java:346)
			at org.eclipse.jetty.http2.HTTP2Connection.onReset(HTTP2Connection.java:259)
			at org.eclipse.jetty.http2.parser.BodyParser.notifyReset(BodyParser.java:139)
			at org.eclipse.jetty.http2.parser.ResetBodyParser.onReset(ResetBodyParser.java:94)
			at org.eclipse.jetty.http2.parser.ResetBodyParser.parse(ResetBodyParser.java:61)
			at org.eclipse.jetty.http2.parser.Parser.parseBody(Parser.java:229)
			at org.eclipse.jetty.http2.parser.Parser.parse(Parser.java:156)
			at org.eclipse.jetty.http2.parser.ServerParser.parse(ServerParser.java:121)
			at org.eclipse.jetty.http2.HTTP2Connection$HTTP2Producer.produce(HTTP2Connection.java:349)
			at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:512)
			at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:258)
			... 6 common frames omitted
			Suppressed: java.lang.NullPointerException: Cannot invoke "org.eclipse.jetty.http.HttpField.getHeader()" because "f" is null
				at org.eclipse.jetty.http.HttpFields$Mutable.remove(HttpFields.java:1437)
				at org.eclipse.jetty.ee10.servlet.ServletContextResponse.resetContent(ServletContextResponse.java:385)
				at org.eclipse.jetty.ee10.servlet.ServletChannel.handle(ServletChannel.java:458)
				at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1292)
				at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.run(ContextHandler.java:1285)
				at org.eclipse.jetty.ee10.servlet.ServletChannelState.runInContext(ServletChannelState.java:1289)
				at org.eclipse.jetty.ee10.servlet.ServletChannelState.asyncError(ServletChannelState.java:858)
				at org.eclipse.jetty.server.handler.ContextHandler$ScopedContext.accept(ContextHandler.java:1273)
				at org.eclipse.jetty.server.handler.ContextRequest.lambda$addFailureListener$1(ContextRequest.java:53)
				at org.eclipse.jetty.server.internal.HttpChannelState.lambda$onFailure$2(HttpChannelState.java:446)
				at org.eclipse.jetty.util.thread.SerializedInvoker$Link.run(SerializedInvoker.java:191)
				at org.eclipse.jetty.http2.server.internal.HttpStreamOverHTTP2.lambda$onFailure$1(HttpStreamOverHTTP2.java:594)
				at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
				at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
				at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
				... 6 common frames omitted

konstantin-mikheev avatar May 13 '24 09:05 konstantin-mikheev

Will close since the loop does not happen in 12.0.9 and there are no 100% CPU usage

konstantin-mikheev avatar May 13 '24 10:05 konstantin-mikheev

The loop got sorted out but the original NPE still exists. I've opened https://github.com/jetty/jetty.project/issues/11785 to follow up on that one.

Again, if you could come up with a reproducer that would be tremendously helpful.

Thanks!

lorban avatar May 13 '24 12:05 lorban