HTTP3/QUIC support

[amyipdev]

What's missing?

As far as I can tell, currently Rocket supports HTTP2 via enabling the HTTP2 feature, but has no support for HTTP3/QUIC. Being able to run dual HTTP3/QUIC would be a great feature for performance, particularly in network-unstable environments.

Ideal Solution

I would recommend something similar to what NGINX does - send the Alt-Svc header for HTTP/1.1 and HTTP/2 requests, take PORTNUM/tcp as HTTP/1.1/2 and PORTNUM/udp as HTTP/3/QUIC. To integrate directly via Hyper, the solutions being worked on at https://github.com/hyperium/hyper/issues/1818 could be helpful.

Why can't this be implemented outside of Rocket?

Currently I reverse proxy all my services, which does let me use QUIC between clients and the reverse proxy. However, that does not work when people are running Rocket directly without a reverse proxy.

Are there workarounds usable today?

There's the previously mentioned reverse proxy solution, but that is not in-Rocket.

Alternative Solutions

No response

Additional Context

No response

System Checks

• [X] I do not believe that this feature can or should be implemented outside of Rocket.
• [X] I was unable to find a previous request for this feature.

[SergioBenitez]

I'd like nothing more than for Rocket to support HTTP/3, but doing so without having support in upstream hyper would require a considerable amount of work.

As far as I can tell, quiche is the only mature HTTP/3 implementation in Rust. It's not clear to me whether the library can correctly be used in an async server, however, as its docs don't state whether methods like poll block.

Aside from quiche, the only other viable alternative I'm aware of is h3, which would become hyper's HTTP/3 backend. According to the README, it is "still very experimental", and the report indicates there's still quite a road to be compliant. Nevertheless, it seems viable to integrate into Rocket as a sort of "draft" or "preview" of HTTP/3 support. If there is some way to integrate h3 into Rocket in such a way that doesn't require drastically changing the existing codebase, I'd be all for it. Otherwise, we'll just have to wait for hyper to gain support or some other library to pop up.

[SergioBenitez]

I wanted to experiment, so I've landed experimental HTTP/3 support based on s2n-quic and a patched version of h3.

It works!

> cd examples/tls
> cargo run
...
🚀 Rocket has launched on https://127.0.0.1:8000 (TLS + MTLS)

curl -k --http3-only -v https://127.0.0.1:8000       master ●
*   Trying 127.0.0.1:8000...
* Skipped certificate verification
* Connected to 127.0.0.1 (127.0.0.1) port 8000
* using HTTP/3
* [HTTP/3] [0] OPENED stream for https://127.0.0.1:8000/
* [HTTP/3] [0] [:method: GET]
* [HTTP/3] [0] [:scheme: https]
* [HTTP/3] [0] [:authority: 127.0.0.1:8000]
* [HTTP/3] [0] [:path: /]
* [HTTP/3] [0] [user-agent: curl/8.6.0]
* [HTTP/3] [0] [accept: */*]
> GET / HTTP/3
> Host: 127.0.0.1:8000
> User-Agent: curl/8.6.0
> Accept: */*
>
< HTTP/3 200
< content-type: text/plain; charset=utf-8
< server: Rocket
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< permissions-policy: interest-cohort=()
< content-length: 13
<
* Connection #0 to host 127.0.0.1 left intact
Hello, world!%

In this h3 branch, only the quic/http3 server runs when the http3 feature is enabled. The idea would be to spawn both the http1/http2 server and the http3 server and respond with an alt-svc: h3 header in the former. Nevertheless, this shows that Rocket's internal architecture allow us to implement and ship some kind of http3 support without too much trouble.

[amyipdev]

@SergioBenitez is that h3 branch something that development should be continued on? I would love to continue work on this.

[SergioBenitez]

@amyipdev Yes. I'll continue to push commits to the branch. I've just now pushed a nearly "complete" version. It's likely that we'll need to rewrite/significantly improve the h3 integration for s2n as well as h3 itself. This is because at the moment, there's no way to get an AsyncRead and AsyncWrite stream to the client once the HTTP3 request has been received. This means we can't directly integrate with the existing Listener/Connection APIs, which means we don't get graceful shutdown for "free".

In short, we need to really implement Listener for some H3Quic type and get rid of Void here:

https://github.com/rwf2/Rocket/blob/b2378ab8e84a73f7620110d6a9f363f76ac9b060/core/lib/src/listener/quic.rs#L84-L107

[amyipdev]

I'll look into it and see if there's any way I can help.

[camshaft]

Please let us (the s2n team) know if you need anything to make the integration easier. Ideally we can get the s2n-quic-h3 crate published so you're not having to maintain a fork.

[amyipdev]

Currently compiling the most recent commit, noticed I had to manually enable rocket_dyn_templates/tera. Didn't come up before - possible regression?

[SergioBenitez]

Currently compiling the most recent commit, noticed I had to manually enable rocket_dyn_templates/tera. Didn't come up before - possible regression?

What are you running? You'll only need to do that if you're trying to compile the dyn_templates crate. If you want to test the crate, run ./script/test.sh. If you want to just work on core, then run your cargo commands inside core/lib.

[SergioBenitez]

@camshaft

Please let us (the s2n team) know if you need anything to make the integration easier. Ideally we can get the s2n-quic-h3 crate published so you're not having to maintain a fork.

Really appreciate the comment! Thank you!

At the moment, aside from https://github.com/aws/s2n-quic/pull/2055, the biggest issues are:

• The server's accept() method requires an &mut reference. This means that we can't accept from multiple threads without synchronization. This is probably fine, but it deviates from existing connection-like interfaces such as TcpListener::accept(). This is also the case on the h3 side, so improving this only on the s2n will unfortunately be insufficient.

• This is mostly on the h3 side, but the lack of Stream and AsyncWrite implementations on the "fully accepted HTTP3 stream." In other words, we'd really like to have a read/write stream to the HTTP3 body once an HTTP3 request has been parsed out, but that's not currently possible because h3::server::RequestStream doesn't implement AsyncRead/AsyncWrite, and an efficient implementation outside of h3 isn't possible due to the use of the Buf trait. We end up with:

  pub struct QuicRx(h3::RequestStream<quic_h3::RecvStream, Bytes>);
  
  pub struct QuicTx(h3::RequestStream<quic_h3::SendStream<Bytes>, Bytes>);
  
  impl Stream for QuicRx {
      type Item = io::Result<Bytes>;
  
      fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
          use bytes::Buf;
  
          match ready!(self.0.poll_recv_data(cx)) {
              Ok(Some(mut buf)) => Poll::Ready(Some(Ok(buf.copy_to_bytes(buf.remaining())))),
              Ok(None) => Poll::Ready(None),
              Err(e) => Poll::Ready(Some(Err(io::Error::other(e)))),
          }
      }
  }
  
  impl AsyncWrite for QuicTx {
      fn poll_write(
          mut self: Pin<&mut Self>,
          cx: &mut Context<'_>,
          buf: &[u8],
      ) -> Poll<io::Result<usize>> {
          let len = buf.len();
          let result = ready!(self.0.poll_send_data(cx, Bytes::copy_from_slice(buf)));
          result.map_err(io::Error::other)?;
          Poll::Ready(Ok(len))
      }
  
      fn poll_flush(self: Pin<&mut Self>, _: &mut Context<'_>) -> Poll<io::Result<()>> {
          Poll::Ready(Ok(()))
      }
  
      fn poll_shutdown(self: Pin<&mut Self>, _: &mut Context<'_>) -> Poll<io::Result<()>> {
          Poll::Ready(Ok(()))
      }
  }
  

  The simplest way to resolve this would likely be for h3 to implement Stream and AsyncWrite if the underlying streams do, which in this case would mean that s2n's RecvStream and SendStream would need the appropriate implementations.

In general, it seems that most if not all of the issues I've encountered are due to h3.

[camshaft]

The server's accept() method requires an &mut reference. This means that we can't accept from multiple threads without synchronization. This is probably fine, but it deviates from existing connection-like interfaces such as TcpListener::accept(). This is also the case on the h3 side, so improving this only on the s2n will unfortunately be insufficient.

Can you open an issue for this? I understand the ask but we'll need to figure what it looks like in practice. This would end up being essentially a "spmc channel". A naive distribution strategy for new connections could just maintain a queue and round robin. But if one of those acceptors is too slow then the load wouldn't be well-distributed and could lead to poor performance.

In general, it seems that most if not all of the issues I've encountered are due to h3.

Yes it's still very much an early implementation... Definitely still some issues with the underlying interface that haven't been solved, too: https://github.com/hyperium/h3/issues/78

[SergioBenitez]

Here's something that might be able to help on the s2n side.

The Listener interface we have is as follows:

1. We start with something we can bind() to. This is usually a SocketAddr, reified as Bindable. Calling bind() returns a Listener.

2. The listener accepts connections in two phases: accept() and connect().

   The contract is such that accept() must do as little work as possible as it is intended to be used in an accept-loop. connect() on the other hand is expected to be called in a separate task and can do whatever work is needed to prepare the connection for reading/writing. This is where the TLS handshake occurs, for example.

3. An HTTP request is read from the connection.

Mapping this to HTTP3/quic with the current implementations is proving to be challenging because accepting a connection and reading an HTTP request is currently interleaved in a difficult-to-compose manner. As it stands, we go quic -> HTTP3 -> connection. In other words, we need to do some work with HTTP3 before we can get our hands on a valid peer connection.

This doesn't seem endemic to the design, however. If we instead reverse the arrows (quic -> connection -> HTTP3), then we recover the previous design. Concretely, this would mean that s2n_quic_h3 has some type T that implements h3::quic::Connection while also being AsyncRead and AsyncWrite. I believe that type is PeerStream.

Do you think it's possible to implement h3::Connection for PeerStream, or some other type that would provide the sought-after behavior?

[SergioBenitez]

The h3 branch now implements complete support for HTTP3. There's still a lot of polish necessary, but I foresee it going into mainline very soon.

[amyipdev]

@SergioBenitez good to hear, keep me posted! And if I can help in any way with polishing please let me know

[SergioBenitez]

Support is ready to land on master! One key missing bit is support for mTLS over QUIC. s2n-quic makes this a bit difficult to do at the moment (https://github.com/aws/s2n-quic/issues/1957). @amyipdev Can you keep tabs on the rustls update in https://github.com/aws/s2n-quic/pull/2143 and mTLS in https://github.com/aws/s2n-quic/pull/2129 and perhaps update our QUIC integration to 1) expose mTLS certificates and 2) not create a second rustls config once https://github.com/aws/s2n-quic/pull/2143 lands?

[amyipdev]

@SergioBenitez I'll do my best!