Investigate on how to use Discovery v5 talkreq/talkresp message packets for creating data streams

[kdeme]

Current specification indicates to use the TalkReq message to send data over an uTP stream: https://github.com/ethereum/stateless-ethereum-specs/blob/master/discv5-utp.md#specification , with no TalkResp message. This actually violates the current Discovery v5 specification.

For now, this route can taken (perhaps just reply with empty TalkResp messages) so we can focus on the rest of the implementation.

However, I believe eventually a "real stream" should be set up, whether as part of the Discovery Protocol (A new packet type) or outside of Discovery Protocol. In the latter case Discovery Talkreq/Talkresp would still be used to initiate the uTP connection_id (if uTP is to be used) and the AES keys negotiated in the Discovery session can still be used for encryption. However, a packet muxing step would have to go in front of the arrival of the data on the socket, before being passed to Discovery. In the end, both solutions are similar, and it is rather about where this packet muxing step is done/checked. One should carefully think about how to mux the data, without having to waste too much on (failed) decryptions. It could be useful also to keep current header obfuscation of discovery v5 against packet inspection.

This issue is about investigating this and perhaps providing a PoC.

[kdeme]

Extra note: To investigate / PoC this, no actual (full) implementation of uTP would be needed

[kdeme]

Some quick thoughts on the options I have in mind:

1. Use the TalkReq message type to encapsulate the uTP stream. As mentioned, this violates the current discv5 specification, as no TalkResp message is send.
   • Can just send an empty TalkResp each time. Or try to adjust discv5 specification.
   • It does mean that each message gets extra overhead because of the discv5 packets + talkreq message.
2. Set up the stream outside of discv5: merely provide the uTP connection_id (or some other identifier, mind you that this stream set-up should be independent of what is actually used) through the talkreq/talkresp message, in the respective Portal message payload. As the same port / socket is to be reused for this stream, there needs to be muxing capability in front of discovery and the uTP stream, to select which type of packet it is.
   1. Basic solution: uTP connection_id prefixed to payload, payload encrypted by same session key as set-up in discv5 (This must have been set up there earlier as it needs to do a successful talkreq/talkresp to get the connection_id). Muxer can first pass the data to the stream, checking if connection_id is known. If not, pass it to discv5. No unnecessary decryption. Not adding theconnection_id however would require to either try the discv5 header decryption or the stream decryption.
   2. When we want to benefit from the (security) properties added thanks to the masked header from discv5: Reuse packet format of discv5, that is masking IV || masked header || payload. This adds overhead of course, but a little less than in solution 1. (where you would also have message type, req id and protocol data from talkreq). You also don't need to add the connection_id as in 2.i., as this identifier is also passed in the uTP header. More importantly it makes a cleaner separation of request- response type of communication versus a data stream. Practically, the protocol field in the discv5 header could be changed to something like "stream" instead of "discv5" to be able to mux this.
3. Same as 2.ii basically, but this time really incorporated in discv5 spec. Would keep the protocol field in discv5 header to "discv5", but the flag field can be set to a new package type, e.g. Stream Packet.

As mentioned, 2.ii. and 3. are very similar. 2.ii. would require probably more architectural changes to be able to nicely separate the stream from the discv5 code (it would rather have to be some sort of a plugin/extension, instead of the muxing in front of protocols as same header decryption needs to be done), but then again 2.ii. requires no changes/additions to the discv5 spec.

But perhaps something similar as 2.i. can also sufficient. In this basic solution the connection_id + encrypted data can look random. However, the connection_id will get repeated over several stream messages. One would also definitively have to check the IP from where the data came. So yeah, perhaps something smarter can be done here still, but you might then eventually arrive again at 2.ii or 3.

All in all, to good thing is that even solution 1.a can be used fine for a PoC for now, as in practise the additional extra overhead is not much (compared to 2.ii. or 3.).

[KonradStaniec]

Use the TalkReq message type to encapsulate the uTP stream. As mentioned, this violates the current discv5 specification, as no TalkResp message is send.

From what I know uTP is reliable steam protocol, therefore for each send packet there will be corresponding ack from remote node. Can't we structure protocol that way, that are outgoing packet will be in TALKREQ and all acks will be in TALKRESP this way each req will always have response.

I am also thinking that we may need some kinda specification, of each stream life cycle i.e who should close it, how each should happen, should the stream hang in after sending response etc.With multiple network there will be a lot of stream between different peers, and it would be good not to have some dangling resources. Maybe we can somehow re-use some parts of mplex spec from libp2p which specifies how to multpliex multiple streams https://github.com/libp2p/specs/blob/master/mplex/README.md?

[kdeme]

Current state:

• Currently uTP is send over TalkReq messages (Option 1 from above comment):
  • Talkresp is send back empty, in order to satisfy discv5 (avoid routing table evictions), but the message is ignored on the Portal network layer.
  • Using TalkResp in some way for uTP acks does not sound like a good plan as you want the bidirectional data flows to be independent from each other (this was mentioned in above comment and on some other occasions).
  • It works.

Recap on other options / directions to go:

• 1.ii: New discv5 message type, "TalkStream" that does not require a response.
  • Could also allow for TalkReq without a TalkResp. But this is troublesome for messages that do require req/resp mechanism. How can base layer still decide when to evict nodes?
• 2.i: Something custom on same UDP port: e.g. connection id prefix + encryption through discv5 session reuse
  • Muxing in front of discv5 is a bit more annoying
  • Need to be careful on holding the session data available (in relation with connection id & IP probably) over the course of uTP connection set-up & use.
  • Could of course use yet another Port, but that is more cumbersome for the user to set up (for sure when behind NAT).
  • Could also use only a different outgoing Port, and filter on that (Not sure, edge cases here?).
  • connection_id -> repeated for each message
  • Not keeping track of Node Id, only have access to IP.
  • The above two items can be resolved by adding more to the packet, but that sounds a lot like just redoing what discv5 already does
• 2.ii: Sounds mostly fine, just when reusing so much of discv5, it will basically end up living in that same code and would be better as a part of the spec, and thus resulting in option 3.
• 3: See above.
• 4: Something not (yet) investigated: other protocols that add similar security properties as discv5 + reliable transfer in one go? E.g. QUIC? (I think the problem with QUIC is mostly the fact that it is will always do encryption, through TLS. Which would be an issue if we'd want to reuse the discv5 session. And if we do not want to do that, then I'm not sure what impact, with regards to certifications and such, this has when setting up connections. Anyway, TBI).

Some numbers on message overhead:

• Discv5 + TalkReq message overhead: 107 bytes overhead (With "utp" as protocol id)
• Discv5 + TalkResp message overhead: 103 bytes
• Option 1: 107 bytes overhead request & 103 bytes overhead response
• Option 1.ii (No reponse): 107 bytes
• Option 2.i: 2 bytes + HMAC = 18 bytes?
• Option 2.ii: 87 bytes
• Option 3: 87 bytes

In general I think we should definitely be able to get rid of the unnecessary talkresp message that is send now. Option 2.i here sounds like giving up on some of the obfuscation, and possibly also introducing some edge cases, which might end up in doing something custom that does the same as discv5 does (but potentially worse). I'm not really in favor of rolling something custom here. I'd be more in favor of option 1.ii, option 2.ii or option 3, which are all very similar (option 2.ii is least fun). But some investigation first on a potential Option 4 should not be skipped though.

One thing to note is that none of the solutions currently consider for muxing and/or connection reuse, as it is unclear how that would work in the current connection setup scheme with FindContent/Content and Offer/Accept.