Handshake failure when setting up multiple connections over same UDP "connection"

[edsko]

(This is an improved version of #49.)

Problem description

I am trying to use the quic library to setup a QUIC server that serves an existing QUIC client, which is not written by me. My server at the moment is extremely simple, merely printing a message when any connection has been setup successfully at all. That message is never printed.

    QUIC.run config $ \conn ->  do
      putStrLn $ "Accepting connection"

When I enable scDebugLog, I'm seeing errors like this

debug: cannot decrypt: InitialLevel size = 1200

Analysis

I've done my best to debug this, and with my limited knowledge of the QUIC protocol and the quic source, this is my best attempt at an analysis:

• In Wireshark I can see that the client is sending a bunch of Initial messages. Indeed, these are the only messages sent from the client to the server; they do not get past this.
• Importantly, those Initial messages are all using the same UDP "connection" (of course UDP is not connection oriented; I merely mean: the same destination IP/port and same source IP/port). That particular setup might seem strange, but for example it is explicitly mentioned in the use cases of the Rust Quinn library for QUIC as a "Single Socket Example", which states "You can have multiple QUIC connections over a single UDP socket", and explains why one might do this.
• The Initial message in the QUIC handshake contains a CRYPTO frame, which is encrypted with a key that is derived from the destination ID used by the client.
• After adding a lot of instrumentation to quic, I can see that the first Initial message that comes in is handled just fine, but the second one is not, and results in that cannot decrypt exception.
• Importantly, the server only prints the debug Original CID once, and indeed, when I place a trace on the initialSecret function, I see it being called only once. What I think is happening therefore is that quic is trying to decrypt that second Initial message using the CID from the first message, which would result in using the wrong key, which would result in that decrypt failure.
• Possibly a related observation (possibly not): if we look at the compatibility matrix, in the entry for "Haskell quic" as server and Quin as client, we find only a "B", missing an "M" for "Server CID change".

For completeness' sake, I've made these packets available at https://github.com/edsko/quic/tree/edsko/quinn-interop-demo/quinn-interop/problem-packets , both the raw packet bytes as well as the full Wireshark decoding.

To reproduce

I have spent a few days trying to come up with a minimal example that reproduces the problem, but I haven't quite succeeded, although I think (I hope) that I've come close.

On that same branch I mentioned above at https://github.com/edsko/quic/tree/edsko/quinn-interop-demo/quinn-interop there is a Haskell project and a Rust project (I cannot demonstrate this with Haskell only, because the abstractions offered by quic do not make it possible to setup multiple connections with the same UDP client port). Instructions are in the README, but briefly, if we run the Haskell server with

cabal run simple_server

and the Rust client with

cargo run --bin concurrent_bi_client -- 127.0.0.1:5001

and look at the debug_log directory, we do see that same decrypt failure in the logs. It's not a precise reproduction of the problem though, because the server seems to be able to recover from the error, and the client does succeed in talking to the server, unlike in the real problem case. However, the decrypt failure does happen, as I mentioned, and moreover, if I look at the first two messages that get sent to the server in Wireshark, they look identical to the actual problem: two Initial messages that differ essentially only in the destination ID (but, importantly, but the same UDP source and destination ports). Again, for completeness' sake I've included these packets in https://github.com/edsko/quic/tree/edsko/quinn-interop-demo/quinn-interop/problem-packets as well.

[kazu-yamamoto]

Sorry for the delay. I did not notice this issue.

QUIC supports multiple connections on a single (IP-ver, dst-addr, src-addr, dst-port, src-port). To distinguish connections, destination connection IDs are used.

Probably, quinn make multiple connections on a single tuple. Unfortunately, Haskell quic assumes one connection per tuple.

Are there any client options to use a single connection per tuple in quinn?

[kazu-yamamoto]

According to https://quinn-rs.github.io/quinn/quinn/set-up-connection.html, probably you can use a different port for each connection.

[edsko]

Thanks for your response! The problem is not quinn per se; like you said, although it supports this workflow, it does not require it. However, the problem is that I am not in control over the client at all; that's an existing application that I need to interact with as-is.

[edsko]

@kazu-yamamoto However, the client for whom I was writing this code has currently suspended the project due to lack of funds, so I am currently not blocked on this (not that you have any obligation to look at it even if I was blocked on it, obviously! :smile: ).

[kazu-yamamoto]

For record, if you use a different port for each client, concurrent_bi_client would work well.