[Tracking Issue] QUIC Transport

[elenaf9]

Summary

Implement the QUIC transport for rust-libp2p.

Done criteria

• Quic can be used as a transport in the Swarm for communication between nodes that can directly reach each other
• TLS handshake is compliant with the libp2p/specs TLS Handshake and unit tested
• Transport is unit & integration tested
• Benchmarked with mxinden/libp2p-perf
• Tested again go-libp2p with Testground
• Public API is documented

Status Quo

Current implementation:

• #2289
  • Working prototype, but with unaddressed reviews, and open TODOs in the code
  • uses quinn-proto quic implementation -> pure state-machine without any I/O or async code

Alternative proposal / draft:

• #2801
  • uses quinn: high level crate; compared to quinn-proto quinn also takes care of I/O and driving the endpoint and connections
  • Depends on unmerged PRs in quinn: quinn-rs/quinn#1219 and quinn-rs/quinn#1357

Previous attempts / History

• #211
  • Initial Issue
• #563
  • First implementation using picoquic
  • Not using quinn because back then most quinn types were !Send
• #862
  • Continuation of #563
• #1334
  • Use quinn-proto
  • Continuation in branch tomaka/quiccc-again
• #2144
  • Inspired by #1334 but does not share code
  • Related repo: https://github.com/ipfs-rust/libp2p-quic
  • published on crates.io as libp2p-quic
  • Support noise in addition to tls
  • Closed because quic+noise not in libp2p specs
• #2159
  • Continuation of #2144
  • Handles ecn codes and gso (throughput)
  • "Support for plugging in your own crypto"
• #2289 [Most recent PR]
  • Initially a continuation of #2159
  • Later code was replaced with old code from tomaka/quiccc-again branch
• #2801 Alternative to #2289
  • Uses quinn instead of quinn-proto

Tasks

• [x] Finish kpp#23 (replaces elenaf9#6)
• [ ] Finish remaining TODOs / FIXMEs in #2289:
  • [ ] Handle / use ECN bits
  • [ ] Review / fix local_address when processing an incoming UDP datagram
• [ ] Evaluate whether we should switch to use UdpSocket from quinn_udp instead of the rust std implementation
• [ ] Review how backpressure is enforced throughout the whole implementation
  • If the application is too busy to read from an existing substream
  • If max inbound substreams is reached
  • If one connection is busy
• [ ] Enhance docs
• [ ] Enhance test
  • [ ] Enhance ping test in libp2p/test-plans for interop tests with Testground
  • [ ] More tests in general: backpressure, closing substreams, closing connections, limits, ..
• [ ] Add QUIC support to mxinden/libp2p-perf for benchmarks

Long-term (not required for first implementation):

• Properly implement Transport::dial_as_listener to support hole-punching
• Handle address change if a connection migrates
• When first starting to listen, enable server-capabilities on existing client-only endpoint instead of creating a new one
• Evaluate alternatives to the quinn_proto quic implementation
  • quinn
  • s2n-quic (AWS)
  • quiche (Cloudflare)

[mxinden]

Thanks for creating this @elenaf9!

Whenever you make a significant change to the original issue description, would you mind posting a comment like https://github.com/libp2p/rust-libp2p/issues/2052#issuecomment-827509552? That way, everyone subscribed to this issue gets a notification.

[elenaf9]

Changelog

Handle address change if a connection migrates

• Moved to Long-term. For now we can simply disable connection migration.

[elenaf9]

Changelog QuicMuxer refactoring (elenaf9#6) was replaced and merged into #2289 with kpp#23:

• Updated the status-quo description
• Marked corresponding task as done.

[elenaf9]

Changelog

Evaluate whether we should switch to use UdpSocket from quinn_udp instead of the rust std implementation

• For the first implementation we will stick with UdpSocket from rust std. Integrating quinn_udp::UdpSocket requires additional complexity, and we don't strictly need it, especially if we plan on long-term switching to quinn.

Handle / use ECN bits

• Moved to Long-term since the UdpSocket from std does not support this.

Fix local_address when processing an UDP datagram for a new incoming connection on a socket that is bound to a wildcard address

• Moved to Long-term. To fix this, we need to obtain the destination-ip of the first udp packet on the connection, which we can not access through the normal UdpSocket API. quinn_udp supports it for linux only. Long-term we can contribute support for other platforms as well (related issue: quinn-rs/quinn#508). For now the local_address will have an unspecified IP in this case.

[marten-seemann]

Should Stateless Reset handling / generation be added to the Long Term list?

[elenaf9]

Changelog:

• [x] Finish remaining TODOs / FIXMEs in Libp2p quic third attempt #2289
• [x] Enhance docs
• [x] Review how backpressure is enforced throughout the whole implementation -> Described in the docs on [endpoint::Driver] in #2289

[thomaseizinger]

Changelog:

Added link to https://github.com/kpp/rust-libp2p/pull/27#discussion_r1018147667 so we don't forget to refactor the TLS-config handling.

[elenaf9]

Changelog:

• First QUIC implementation based on quinn proto is merged with #2289 :tada:. Updated Status Quo description.
• Add new task for support the quic codepoint / draft-29. #2289 only added support for quic-v1. See https://github.com/multiformats/multiaddr/issues/145.

[geotro]

I hope this is the right place to ask.

Is quic supported on relay-v2, and if not, what are the future plans regarding this?

[elenaf9]

@geotro yes it is supported. You can combine them in the same manner as it is done here with tcp and quic: https://github.com/libp2p/rust-libp2p/blob/a7148648858fe10e9ba4c2793c7e12392b49c0ab/transports/quic/tests/smoke.rs#L118-L136

and either replace the tcp_transport with your relay transport, or if you'd like to support tcp as well you have to combine (OrTransport) tcp and relay and then do the upgrade.

Does this help?

[elenaf9]

Changelog

• Add link to libp2p/rust-libp2p#3133

[geotro]

@elenaf9 it helps a bit, but I was hoping that once QUIC is implemented to be able to do UDP, as well as DCUTR.

Why are you referring to TCP in the context of QUIC?

I understand DCUTR with QUIC is not yet supported so I'll have to wait until that works before I retry it.

[elenaf9]

Why are you referring to TCP in the context of QUIC?

I was referring to TCP to point out how you could support both TCP and QUIC together with the relay-v2 protocol. But TCP is not required, you may as well just combine QUIC and relay-v2.

[geotro]

Ah OK, thank you for clarifying this!

My use case requires DCUTR so I'm going to have to wait until this is available together with QUIC before I can begin testing/using it. Very much looking forward to this as I hope it's going to greatly improve the success rates of establishing a direct connection!

[Frando]

Is there an issue or note somewhere what is missing for a working setup with QUIC + DCUTR + relay v2? Looking what pieces are missing there and would love any pointers.

[mxinden]

Is there an issue or note somewhere what is missing for a working setup with QUIC + DCUTR + relay v2?

As far as I remember, the only thing missing is the implementation of <quic::GenTransport as Transport>::dial_as_listener.

https://github.com/libp2p/rust-libp2p/blob/56b3b8fe5a1e5d0b14ba7987a74ff3bc29f6b4e2/transports/quic/src/transport.rs#L182-L191

Note that https://github.com/libp2p/punchr/blob/main/rust-client/ is already using QUIC for hole punching, though only on the Transport::dial and not on the Transport::dial_as_listener side.

@Frando contributions here are welcome.

[mxinden]

Posting some in-progress research on the various Rust QUIC implementations out there.

Verdict

Thus far my preference is still quinn via https://github.com/libp2p/rust-libp2p/pull/3454.

• We know at least quinn-proto works well for us (see libp2p-quic alpha release).
• It provides all the features we need (see table below).
• From what I can tell it has the most promising in-progress WebTransport implementation, see WebTransport column below.
• Thus far, unable to spot any performance issues via https://github.com/libp2p/test-plans/pull/163/, see https://observablehq.com/@mxinden-workspace/libp2p-perf using quinn.

With https://github.com/libp2p/rust-libp2p/pull/3454, quinn is no longer exposed on the public API surface. Thus we can easily switch out the underlying QUIC implementation in the future in case we ever want to move off of quinn.

Next step: Report our performance measurements to quinn maintainers to push https://github.com/quinn-rs/quinn/pull/1219 forward.

Comparison

	integration effort	comments	WebTransport	Pacing	Managed UDP socket	GSO GRO	random bytes for hole-punching	TLS	license
quinn	low, see PR 3454	Uses unbounded channels	Work-in-progress	Pacing implementation	Yes, through quinn crate	Added with GSO GRO	Yes. One can gain access to the socket via Endpoint::new	rustls and ring	Apache and MIT
quiche	unknown	Large cooperation focused on performance (Cloudflare)	Work-in-progress out of tree community owned	Pacing implementation	No, see example on manual management	Doesn't manage socket, thus only example	Yes, since UDP socket isn't managed	boring ssl	BSD 2-Clause
s2n	unknown	Large cooperation focused on performance (AWS) \n use unbounded channel to accept new connections. Needs to be investigated whether that is a problem.	Can't find any	yes	Yes, see client and server	yes	Not sure how to access the socket on a `Server`.	s2n-tls	Apache
msquic	unknown	API for Rust, published on crates.io
neqo	unknown	not published on crates.io

//CC @marten-seemann

[marten-seemann]

Thanks for compiling this list @mxinden! This looks pretty comprehensive.

Some thoughts:

• Not sure if GSO and pacing warrant their own column, they’re just part of a long list of things a QUIC implementation can (optionally) do to speed things up. Others would be DPLPMTUD and ECN. They might serve as a proxy variable to see how advanced a QUIC stack is though.
• I expect WebTransport to be supported by more implementations once the draft makes more progress at the IETF. Not a lot of implementations are willing to pay the cost of shooting at a moving target, unless they have a very clear use case for the protocol (as quic-go obviously has, and Chrome’s implementation as well). This is even more so true after the adoption of the RELIABLE_RESET_STREAM draft at the last IETF meeting.
• Choosing Quinn (for now) doesn’t seem like a terrible choice, especially since the work is mostly driven by the community. Making the package API agnostic to the underlying QUIC implementation seems like a wise choice, enabling more experimentation in the future.

[kpp]

Column random bytes for hole-punching quinn - Yes. One can gain access to the socket via Endpoint::new

The way it is implemented in https://github.com/libp2p/rust-libp2p/pull/3964 is through channels sending quinn_proto::Transmit. However the go impl uses multiple sockets with SO_REUSEPORT. So in quinn wrapper we can go both ways not sure which one is better.

[mxinden]

Status Update

https://github.com/libp2p/rust-libp2p/pull/3454 merged and thus our libp2p-quic implementation depends on upstream quinn directly. Thank you @kpp for the work. In case no major blockers are discovered in libp2p-quic v0.9.0-alpha we can release it as v0.9.0 soonish.

[thomaseizinger]

Status Update

#3454 merged and thus our libp2p-quic implementation depends on upstream quinn directly. Thank you @kpp for the work. In case no major blockers are discovered in libp2p-quic v0.9.0-alpha we can release it as v0.9.0 soonish.

It appears that the above description of this issue is rather outdated, do you mind updating it?

[mxinden]

Status Update

We just released libp2p v0.52.3 which contains a stable release of libp2p-quic v0.9.2. That is the first stable libp2p-quic release in rust-libp2p and thus a big step for the project :tada:

[mxinden]

I am closing here as the major milestone, adding QUIC to rust-libp2p, is achieved. Smaller follow-up improvements are tracked in individual issues.

Thank you to the many many many people involved in this effort. :heart: