Suggestions for quinn-udp

[larseggert]

A few suggestions that would IMO make quinn-udp more useful, in no particular order:

• Conversion from EcnCodepoint into u8.
• Setting/getting IPv4 DSCP (IPv6 traffic class) info, i.e., the other six bits not used by ECN.
  • With similar u8 conversion for the whole field and the ECN/DSCP components individually.
• Setting/getting IPv4 TTL (IPv6 hop count) information on packets.
• Pacing offload via GSO (see quiche)

(NB: Rust newbie here, so some of these things might be doable already, but I couldn't quite figure out how).

[Ralith]

Conversion from EcnCodepoint into u8.

x as u8 will do this alrady, though the expectation is that you shouldn't need to do that yourself, since quinn-udp's responsible for passing it down

Setting/getting IPv4 DSCP (IPv6 traffic class) info, i.e., the other six bits not used by ECN.

This would be a straightforward extension. Any thoughts on how this should be exposed to the application layer by a QUIC impl?

Setting/getting IPv4 TTL (IPv6 hop count) information on packets.

Is this relevant to QUIC? We've been reluctant to let quinn-udp scope-creep into a generic UDP library.

Pacing offload via GSO (see quiche)

Pacing offload be cool to have, but seems pretty niche because it requires manual configuration of the kernel to be effective, so won't be useful for end users/unsophisticated deployments. Is there an easy way to automatically detect when it's available? Also, you say "via GSO", but it looks to me like this entire GSO batch is sent with a single txtime there?

[larseggert]

x as u8 will do this alrady, though the expectation is that you shouldn't need to do that yourself, since quinn-udp's responsible for passing it down

Doh. Of course.

Setting/getting IPv4 DSCP (IPv6 traffic class) info, i.e., the other six bits not used by ECN.

This would be a straightforward extension. Any thoughts on how this should be exposed to the application layer by a QUIC impl?

"It depends." Some DSCPs such as LE have straightforward semantics for an entire connection. Others may be better used for streams or individual QUIC datagrams (modulo the overall uncertainty if the network is equipped to deal with intra-flow DSCP changes.)

My main motivation here is to enable experimentation with DSCP and QUIC, which obviously needs the UDP I/O to allow this. IMO DSCP on the Internet was held back for decades because it is difficult to use from an application.

(Same is true for the IPv6 flow label. I didn't include it in the list in the first post, but if I ask for a pony, why not ask for a pink one...)

Setting/getting IPv4 TTL (IPv6 hop count) information on packets.

Is this relevant to QUIC? We've been reluctant to let quinn-udp scope-creep into a generic UDP library.

Understood. I really wish for a generic UDP I/O crate though :-)

One use of TTL in QUIC would be to take a TTL change on a path as an indication of a path (routing) change, an possibly do something congestion-control-related. Again, this would be for experimentation at this time, but again, the UDP I/O layer needs to support it to enable this kind of experimentation.

Pacing offload via GSO (see quiche)

Pacing offload be cool to have, but seems pretty niche because it requires manual configuration of the kernel to be effective, so won't be useful for end users/unsophisticated deployments. Is there an easy way to automatically detect when it's available? Also, you say "via GSO", but it looks to me like this entire GSO batch is sent with a single txtime there?

I might have misunderstood what quiche is actually doing, I just took a cursory glance. I had assumed that some of the concepts in this talk were available now in Linux and should hence be used. But I didn't actually check on the merge status of the associated PRs.

[djc]

Setting/getting IPv4 TTL (IPv6 hop count) information on packets.

Is this relevant to QUIC? We've been reluctant to let quinn-udp scope-creep into a generic UDP library.

I think I'd be open to some scope creep assuming there's not too much complexity, and I think we'll probably only passively maintain that stuff (that is, review incoming PRs). It seems wasteful if we're maintaining something that is the best UDP crate to require others to fork it on account of adding some (presumably small things)?

[larseggert]

The recent change to just rip out sendmmsg/recvmmsg support makes me hesitant to depend on quinn-udp for neqo/Firefox. I think quinn-udp needs to decide if it wants to be a generic UDP Rust crate, or whether it wants to cater to quinn only (which would be a totally valid choice). If quinn-udp wants usage beyond quinn, it's feature set needs to be managed more carefully.

[djc]

The recent change to just rip out sendmmsg/recvmmsg support makes me hesitant to depend on quinn-udp for neqo/Firefox. I think quinn-udp needs to decide if it wants to be a generic UDP Rust crate, or whether it wants to cater to quinn only (which would be a totally valid choice). If quinn-udp wants usage beyond quinn, it's feature set needs to be managed more carefully.

How does sendmmsg/recvmmsg support affect its feature set? As long as it correctly and efficiently funnels UDP packets across the network, why do you care what underlying API it uses?

(For reference: https://github.com/quinn-rs/quinn/pull/1729.)

[larseggert]

Well, "feature" as in "provides a moderate performance benefit on POSIX platforms".

[Ralith]

The recent change to just rip out sendmmsg/recvmmsg support

recvmmsg support was not removed, and we have no plans to remove it. sendmmsg was removed because, as far as we can tell, it's not actually helpful in practice, at least when GSO is available. If you have a use case where it makes sense I'd be interested in hearing details.

I think quinn-udp needs to decide if it wants to be a generic UDP Rust crate, or whether it wants to cater to quinn only (which would be a totally valid choice).

quinn-udp is expressly focused on Quinn's requirements, and on things useful to QUIC implementations in general insofar as that Quinn might plausibly benefit from them in the future. As djc mentioned in https://github.com/quinn-rs/quinn/issues/1749#issuecomment-1905816801 there's some wiggle room around stuff that would impose insignificant runtime and maintenance overhead, which we can evaluate on a case by case basis.

I think features of interest to neqo/Firefox are likely to align well with our interests, so we'd be happy to hear feedback about changes that affect you.

[larseggert]

sendmmsg was removed because, as far as we can tell, it's not actually helpful in practice, at least when GSO is available. If you have a use case where it makes sense I'd be interested in hearing details.

I don't think GSO support is prevalent enough on the platforms/versions we need to support that we can solely rely on it.

[Ralith]

Which of those platforms/versions don't support GSO? Some incremental work in quinn-udp may be needed in places (e.g. https://github.com/quinn-rs/quinn/issues/1554, and I think we need to relax the max GSO segment limit for *BSD) but I think every tier 1 kernel and most tier 2 have the capability. Not sure which Windows/macOS versions first introduced support.

[mxinden]

Following up on the removal of sendmmsg in https://github.com/quinn-rs/quinn/commit/ee0882657a16a8ffd7ff5844f355caca519e63ce and its implication for Firefox. Summarizing here since it lead to confusion elsewhere pointing to this discussion.

Long story short, I don't think sendmmsg is relevant for Firefox.

As far as I can tell, Firefox roughly creates one UDP socket per QUIC connection.

GSO/GRO is widely available on Linux and Windows.

• Linux
  • GSO >= 2.6.18 https://wiki.linuxfoundation.org/networking/gso
  • GRO >= 2.6.29 https://lwn.net/Articles/358910/
• Windows ~10 https://learn.microsoft.com/en-us/windows/win32/api/ws2tcpip/nf-ws2tcpip-wsasetudpsendmessagesize

Given one UDP socket per connection, each write and read would use the same 5 tuple. Using the same 5 tuple allows the write and read to happen via GSO/GRO, no need for sendmmsg/recvmmsg. (All under the assumption that there is not much to gain from using both sendmmsg AND GSO (recvmmsg AND GRO respectively).)

With the above, let me summarize the relevant APIs per Firefox tier 1 build target:

• Linux / Android (<= 5.0, i.e. kernel 3.16)
  • sendmsg with GSO
  • recvmmsg with GRO
• Windows
  • WSASendMsg with GSO
  • WSARecvMsg with GRO
• MacOS
  • sendmsg
  • recvmsg
  • Worth exploring is sendmsg_x, recvmsg_x` and network.framework

[Ralith]

Thanks for the analysis! That matches my understanding well.

recvmmsg AND GRO respectively

As a minor aside, we do actually do this, but only because it's easy, both in terms of API and to use in a way that could conceivably yield multiple batches on a socket shared between connections.

[larseggert]

OBE