Should servers be emitting many HTTP/2 control frames?

[kevink-sq]

Jetty Version 10.0.26+

Java Version Java 21

Question We are running into an issue where upgrading to Jetty 10.0.26 results in some 500s because the MadeYouReset CVE-2025-5115 mitigation (pr) now includes server side control frames in the WindowRateControl limiter and is dropping HTTP/2 connections.

We manually increased the RateControl limit from 128 -> 1000 but we're still seeing connections dropped for some of our high QPS services. Whereas before 10.0.26, events per second was less than 5~50, it now jumps to 1000+.

My question is, is it expected for the server to be sending this many HTTP/2 control frames per second? It seems like this behavior was always present but is now becoming a problem with the rate limiter. In what scenarios would the server be sending the RST_STREAM control frame?

[sbordet]

A server is not supposed to send many RST_STREAM frames.

If it does, it is likely an application mistake, either the client app resetting, or the client app sending bad headers, or the server app not reading the request content, or the server app sending bad headers, etc.

Jetty 10 is EOL:

• #10485.

Update to Jetty 12.1.x.

[kevink-sq]

Could client behavior be triggering the rate limiter here? Assuming the client is not malicious and trying to invoke MadeYouReset.

I'm thinking this is server initiated given the rate limiter tips over when including server events.

[sbordet]

The code you linked in the last comment is reset send, so if the client sends a lot of resets it will close its own connection.

See here for troubleshooting: https://jetty.org/docs/jetty/12.1/programming-guide/client/http2.html#listeners

[kevink-sq]

Hmm based on the stack trace though does it look like the server is sending the reset when trying to complete the request and finds there is still data on the stream?

exception dispatching to <removed>
org.eclipse.jetty.io.EofException: null
	at org.eclipse.jetty.http2.server.HTTP2ServerConnectionFactory$HTTPServerSessionListener.onFailure(HTTP2ServerConnectionFactory.java:171)
	at org.eclipse.jetty.http2.HTTP2Stream.notifyFailure(HTTP2Stream.java:890)
	at org.eclipse.jetty.http2.HTTP2Stream.onFailure(HTTP2Stream.java:589)
	at org.eclipse.jetty.http2.HTTP2Stream.process(HTTP2Stream.java:407)
	at org.eclipse.jetty.http2.HTTP2Session.failStream(HTTP2Session.java:637)
	at org.eclipse.jetty.http2.HTTP2Session.failStreams(HTTP2Session.java:629)
	at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onSessionFailure(HTTP2Session.java:2043)
	at org.eclipse.jetty.http2.HTTP2Session.onSessionFailure(HTTP2Session.java:587)
	at org.eclipse.jetty.http2.HTTP2Session.onConnectionFailure(HTTP2Session.java:582)
	at org.eclipse.jetty.http2.HTTP2Session.reset(HTTP2Session.java:728)
	at org.eclipse.jetty.http2.HTTP2Stream.reset(HTTP2Stream.java:172)
	at org.eclipse.jetty.http2.server.HttpTransportOverHTTP2.onCompleted(HttpTransportOverHTTP2.java:375)
	at org.eclipse.jetty.server.HttpChannel.onCompleted(HttpChannel.java:959)
	at org.eclipse.jetty.http2.server.HTTP2ServerConnection$ServerHttpChannelOverHTTP2.onCompleted(HTTP2ServerConnection.java:350)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:489)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
Caused by: java.io.IOException: cancel_stream_error/invalid_rst_stream_frame_rate
	at org.eclipse.jetty.http2.HTTP2Session.toFailure(HTTP2Session.java:642)
	at org.eclipse.jetty.http2.HTTP2Session.failStreams(HTTP2Session.java:620)
	at org.eclipse.jetty.http2.HTTP2Session$StreamsState.onSessionFailure(HTTP2Session.java:2043)
	at org.eclipse.jetty.http2.HTTP2Session.onSessionFailure(HTTP2Session.java:587)
	at org.eclipse.jetty.http2.HTTP2Session.onConnectionFailure(HTTP2Session.java:582)
	at org.eclipse.jetty.http2.HTTP2Session.reset(HTTP2Session.java:728)
	at org.eclipse.jetty.http2.HTTP2Stream.reset(HTTP2Stream.java:172)
	at org.eclipse.jetty.http2.server.HttpTransportOverHTTP2.onCompleted(HttpTransportOverHTTP2.java:375)
	at org.eclipse.jetty.server.HttpChannel.onCompleted(HttpChannel.java:959)
	at org.eclipse.jetty.http2.server.HTTP2ServerConnection$ServerHttpChannelOverHTTP2.onCompleted(HTTP2ServerConnection.java:350)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:489)
	at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:461)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:193)
	at org.eclipse.jetty.http2.HTTP2Connection.produce(HTTP2Connection.java:208)
	at org.eclipse.jetty.http2.HTTP2Connection.onFillable(HTTP2Connection.java:155)
	at org.eclipse.jetty.http2.HTTP2Connection$FillableCallback.succeeded(HTTP2Connection.java:450)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)

[sbordet]

Yes, this is normal behavior. Your application should read the data to avoid this.

[Hexcles]

Yes, this is normal behavior

Is this a new behavior in 10.0.26 though?

Your application should read the data to avoid this

By application do you mean the server? To clarify, do you mean the server is meant to read the remaining data on the stream before trying to close/reset it?

[sbordet]

Jetty 10.0.26 contains the fix for MadeYouReset, so yes it may be emitting more resets than previous versions.

By application do you mean the server? To clarify, do you mean the server is meant to read the remaining data on the stream before trying to close/reset it?

I mean your web application should read the request content.

If it does not, Jetty tries its best to read the request content, but if it cannot, it will send a reset to the client indicating that the request has been canceled and the client does not need to send the remaining of the content.