Elaborate on the streaming capabilities of RLPx and devp2p

[dryajov]

Reposting from https://github.com/ethereum/devp2p/issues/71#issuecomment-483364828.

do any of the layer of the RLPx/devp2p stack break the streaming assumptions or not?

• RLP encoding, is it streaming and if so does it propagate all the way down to the - protocol/transport
• eth/X application level protocols, which messages are/should be streamable
• compression?
• encryption?
• muxing
• transport, etc...

Some benefits of streaming are:

• lower memory footprint
  • no need to fetch all data ahead of time before sending out, this is important when large amounts of data need to be fetched from some sore (fs or local db).
• potential to decrease on the wire traffic
  • e.g. if a client requests some data, but midway thru it realizes that the data is stale, it can end the operation without waiting for the full sequence to be transmitted

In general, a stack that allows and encourages streaming fits really well with the overall requirements of Ethereum - i.e. sending sequences of blocks, or chunks of state are a natural fit for that. I believe this approach has great optimization potential, specially when viewed in the context of fast/warp sync.

[jlokier]

RLP blobs and lists encode the length in bytes at the start of the list. In general, this makes it impossible to stream a request or response because the first few bytes to transmit are not known until the end of the data.

The encoding would have to change to something a little different from RLP to enable streaming.

A slight workaround which reduces the round trip time (RTT) is to pipeline multiple requests for smaller amounts of data, which lets the sender break up its large requests, and the responder reply with those smaller data sets.

It's beneficial to pipeline anyway so that the gap between a reply and the next request is not "wasted". Pipelining a bit more adds framing and compression/encryption overhead, but breaks the flow up to reduce the observable RTT, and allows high priority messages to be inserted during those flows.