jetty.project
jetty.project copied to clipboard
jetty
HTTP requests with large headers are not fully processes and timeout
Jetty version(s) Tested on 12.0.9 and 12.0.5
Jetty Environment core
Java version/vendor (use: java -version)
$ java --version
openjdk 21.0.2 2024-01-16
OS type/version
macOS (darwin kernel 23.3.0)
Description
When I create a Jetty server using Jetty 12 when adding support for larger request header sizes. When a request reaches around this size the server doesn't process the entire request and keeps the connection open. After 30 seconds it times out.
Either the request headers are within the limit and should be accepted, or they are not and should return a 431 error.
How to reproduce?
We can make a request to one of the example jetty projects to reproduce.
Setup:
$ git clone https://github.com/jetty/jetty-examples.git
$ cd jetty-examples
$ git checkout 429396f # latest commit at time of writing
# edit embedded/connectors/src/main/java/examples/ServerConnectorHttps.java
# add "httpsConf.setRequestHeaderSize(128 * 1024);" on line 42
# run ServerConnectorHttps (embedded/connectors/src/main/java/examples/ServerConnectorHttps.java
Perform request against the server:
$ curl https://localhost:8443/ -k -H "Test: $(python3 -c 'print("." * 330972)')" -v
curl: (52) Empty reply from server
After 30 seconds request will timeout.
I've spent a little bit of time debugging this issue, and from what I can tell the HttpParser isn't the cause and I suspect somewhere before the HttpParser this is occurring.
I have written a test case for HttpParser which rules out the parser by matching the input and buffering. However one difference is that the ByteBuffer used in tests differs from the NIO buffers used when running the full server.
For posterity here is a test that shows HttpParser appears to process the request in this form which makes me suspect elsewhere.
diff --git a/jetty-core/jetty-http/src/test/java/org/eclipse/jetty/http/HttpParserTest.java b/jetty-core/jetty-http/src/test/java/org/eclipse/jetty/http/HttpParserTest.java
index 57db19b528..88bce2d734 100644
--- a/jetty-core/jetty-http/src/test/java/org/eclipse/jetty/http/HttpParserTest.java
+++ b/jetty-core/jetty-http/src/test/java/org/eclipse/jetty/http/HttpParserTest.java
@@ -306,6 +307,79 @@ public class HttpParserTest
assertEquals(-1, _headers);
}
+ void parseBuffer(ByteBuffer buffer, HttpParser parser) {
+ int remaining = buffer.remaining();
+ while (!parser.isState(State.END) && remaining > 0)
+ {
+ int wasRemaining = remaining;
+ parser.parseNext(buffer);
+ remaining = buffer.remaining();
+ if (remaining == wasRemaining)
+ break;
+ }
+ }
+
+ @Test
+ public void test()
+ {
+ // 16384 for each buffer
+ ByteBuffer buffer1 = BufferUtil.toBuffer(
+ "GET / HTTP/1.1\r\n" +
+ "Host: localhost:8443\r\n" +
+ "User-Agent: curl/8.4.0\r\n" +
+ "Accept: */*\r\n" +
+ "Foo: " + ".".repeat(16304));
+ ByteBuffer buffer2 = BufferUtil.toBuffer(".".repeat(16384));
+ ByteBuffer buffer3 = BufferUtil.toBuffer(".".repeat(16384));
+ ByteBuffer buffer4 = BufferUtil.toBuffer(".".repeat(16384));
+ ByteBuffer buffer5 = BufferUtil.toBuffer(".".repeat(16384));
+ ByteBuffer buffer6 = BufferUtil.toBuffer(".".repeat(16384));
+ ByteBuffer buffer7 = BufferUtil.toBuffer(".".repeat(16384));
+ ByteBuffer buffer8 = BufferUtil.toBuffer(".".repeat(16363) +
+ "\r\n" + "F: a\r\n" + "Connection: ");
+ //130972
+ ByteBuffer buffer10 = BufferUtil.toBuffer(
+ "close\r\n\r\n");
+
+ HttpParser.RequestHandler handler = new Handler();
+ HttpParser parser = new HttpParser(handler, 128 * 1024);
+
+ if (parser.isState(State.END))
+ parser.reset();
+ if (!parser.isState(State.START))
+ throw new IllegalStateException("!START");
+
+ parseBuffer(buffer1, parser);
+ parseBuffer(buffer2, parser);
+ parseBuffer(buffer3, parser);
+ parseBuffer(buffer4, parser);
+ parseBuffer(buffer5, parser);
+ parseBuffer(buffer6, parser);
+ parseBuffer(buffer7, parser);
+ parseBuffer(buffer8, parser);
+ parseBuffer(buffer10, parser);
+
+ assertTrue(_headerCompleted);
+ assertTrue(_messageCompleted);
+ assertEquals("GET", _methodOrVersion);
+ assertEquals("/", _uriOrStatus);
+ assertEquals("HTTP/1.1", _versionOrReason);
+ assertEquals("Host", _hdr[0]);
+ assertEquals("localhost:8443", _val[0]);
+ assertEquals("User-Agent", _hdr[1]);
+ assertEquals("curl/8.4.0", _val[1]);
+ assertEquals("Accept", _hdr[2]);
+ assertEquals("*/*", _val[2]);
+ assertEquals("Foo", _hdr[3]);
+ assertEquals(130971, _val[3].length());
+ assertEquals("F", _hdr[4]);
+ assertEquals("a", _val[4]);
+ assertEquals("Connection", _hdr[5]);
+ assertEquals("close", _val[5]);
+ assertEquals(5, _headers);
+ }
+
@ParameterizedTest
@ValueSource(strings = {"\r\n", "\n"})
public void testSimple(String eoln)
I cannot reproduce this with 12.0.13, can you please upgrade?
Configuring HttpConfiguration.setRequestHeaderSize(128) and starting a server results in:
curl http://localhost:36133/ -k -H "Test: $(python3 -c 'print("." * 200)')" -v
* Host localhost:36133 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying [::1]:36133...
* Connected to localhost (::1) port 36133
> GET / HTTP/1.1
> Host: localhost:36133
> User-Agent: curl/8.5.0
> Accept: */*
> Test: ........................................................................................................................................................................................................
>
< HTTP/1.1 431 Request Header Fields Too Large
< Server: Jetty(12.0.14-SNAPSHOT)
< Date: Wed, 25 Sep 2024 17:07:51 GMT
< Cache-Control: must-revalidate,no-cache,no-store
< Content-Type: text/html;charset=iso-8859-1
< Content-Length: 467
< Connection: close
<
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 431 Request Header Fields Too Large</title>
</head>
<body>
<h2>HTTP ERROR 431 Request Header Fields Too Large</h2>
<table>
<tr><th>URI:</th><td>/</td></tr>
<tr><th>STATUS:</th><td>431</td></tr>
<tr><th>MESSAGE:</th><td>Request Header Fields Too Large</td></tr>
</table>
<hr/><a href="https://jetty.org/">Powered by Jetty:// 12.0.14-SNAPSHOT</a><hr/>
</body>
</html>
* Closing connection
Closing the issue as "works for me".
@kylef please re-open or comment if you still have problems.
@sbordet The example you have shown of this working is using an order of magnitude smaller headers that what I have placed in the steps to reproduce. Small headers that are 200 bytes are completely fine, Jetty only hangs when the headers are significantly larger.
The steps below can still reproduce the issue when using jetty-examples@c119046 which references Jetty 12.0.13. It appears that a header value of 130988 bytes seems to be the exact amount of bytes where this starts timing out. 130988 leads to timeout, 130987 returns 400 error. Somewhere around 130KB header size leads to timeout.
The issue does not appear to be when the header value itself is over this limit, if I make the header field name larger it can also be reproduced. Thus other headers that curl sends may be a factor in being unable to reproduce.
$ curl https://localhost:8443/ -k -H "Test: $(python3 -c 'print("." * 130987)')" -v
...
< HTTP/1.1 400 Bad Request
< Server: Jetty(12.0.13)
< Date: Mon, 30 Sep 2024 14:14:52 GMT
< Cache-Control: must-revalidate,no-cache,no-store
< Content-Type: text/html;charset=iso-8859-1
< Content-Length: 420
...
$ curl https://localhost:8443/ -k -H "Test: $(python3 -c 'print("." * 130988)')" -v
# timeout
How to reproduce?
We can make a request to one of the example jetty projects to reproduce.
Setup:
$ git clone https://github.com/jetty/jetty-examples.git $ cd jetty-examples $ git checkout 429396f # latest commit at time of writing # edit embedded/connectors/src/main/java/examples/ServerConnectorHttps.java # add "httpsConf.setRequestHeaderSize(128 * 1024);" on line 42 # run ServerConnectorHttps (embedded/connectors/src/main/java/examples/ServerConnectorHttps.javaPerform request against the server:
$ curl https://localhost:8443/ -k -H "Test: $(python3 -c 'print("." * 330972)')" -v curl: (52) Empty reply from server After 30 seconds request will timeout.
When not using SSL (on Jetty 12.0.13) ...
$ curl --http1.1 http://localhost:8080/ -k -H "Test: $(python3 -c 'print("." * 130987)')" -v
* Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: */*
> Test: ...........................................(snip)......
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 431 Request Header Fields Too Large
< Server: Jetty(12.0.13)
< Cache-Control: must-revalidate,no-cache,no-store
< Content-Type: text/html;charset=iso-8859-1
< Content-Length: 458
< Connection: close
<
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 431 Request Header Fields Too Large</title>
</head>
<body>
<h2>HTTP ERROR 431 Request Header Fields Too Large</h2>
<table>
<tr><th>URI:</th><td>/</td></tr>
<tr><th>STATUS:</th><td>431</td></tr>
<tr><th>MESSAGE:</th><td>Request Header Fields Too Large</td></tr>
</table>
<hr/><a href="https://jetty.org/">Powered by Jetty:// 12.0.13</a><hr/>
</body>
</html>
* Closing connection 0
This works, on Jetty 12.0.13.
When using SSL on the same version of Jetty ...
$ curl --http1.1 https://localhost:8443/ -k -H "Test: $(python3 -c 'print("." * 130987)')" -vvv
* Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=NE; L=Omaha; O=Webtide; OU=Jetty; CN=localhost
* start date: Sep 29 14:29:14 2024 GMT
* expire date: Sep 29 14:29:14 2025 GMT
* issuer: C=US; ST=NE; L=Omaha; O=Webtide; OU=Jetty; CN=localhost
* SSL certificate verify result: self-signed certificate (18), continuing anyway.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/7.81.0
> Accept: */*
> Test: ..........................(snip).....
TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 431 Request Header Fields Too Large
< Server: Jetty(12.0.13)
< Cache-Control: must-revalidate,no-cache,no-store
< Content-Type: text/html;charset=iso-8859-1
< Content-Length: 458
< Connection: close
<
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 431 Request Header Fields Too Large</title>
</head>
<body>
<h2>HTTP ERROR 431 Request Header Fields Too Large</h2>
<table>
<tr><th>URI:</th><td>/</td></tr>
<tr><th>STATUS:</th><td>431</td></tr>
<tr><th>MESSAGE:</th><td>Request Header Fields Too Large</td></tr>
</table>
<hr/><a href="https://jetty.org/">Powered by Jetty:// 12.0.13</a><hr/>
</body>
</html>
* we are done reading and this is set to close, stop send
* Closing connection 0
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS alert, close notify (256):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
The above was against a standalone Jetty 12.0.13 with only --add-modules=http,https,test-keystore
$ java -jar ../../jetty-home-12.0.13/start.jar
2024-09-30 09:29:14.098:WARN :oejk.KeystoreGenerator:main: Generating Test Keystore: DO NOT USE IN PRODUCTION!
2024-09-30 09:29:14.552:INFO :oejs.Server:main: jetty-12.0.13; built: 2024-09-03T03:04:05.240Z; git: 816018a420329c1cacd4116799cda8c8c60a57cd; jvm 17.0.11+9
2024-09-30 09:29:14.590:INFO :oejus.SslContextFactory:main: x509=X509@5ef5c734(jetty-test-keystore,h=[localhost],a=[],w=[]) for Server@d771cc9[provider=null,keyStore=file:///home/joakim/code/jetty/distros/bases/empty/etc/test-keystore.p12,trustStore=null]
2024-09-30 09:29:14.661:INFO :oejs.AbstractConnector:main: Started ServerConnector@1941a8ff{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}
2024-09-30 09:29:14.668:INFO :oejs.AbstractConnector:main: Started ServerConnector@1a760689{SSL, (ssl, http/1.1)}{0.0.0.0:8443}
2024-09-30 09:29:14.676:INFO :oejs.Server:main: Started oejs.Server@5a5338df{STARTING}[12.0.13,sto=5000] @1469ms
2024-09-30 09:29:24.208:WARN :oejh.HttpParser:qtp525571-59: Header is too large 8193>8192
2024-09-30 09:31:13.155:WARN :oejh.HttpParser:qtp525571-44: Header is too large 8193>8192
Both cases were detected and responded to with Header is too large.