state-history is not fragmenting huge websocket messages

[igorls]

trafficstars

Nodeos 2.1 state-history plugin is not splitting the websocket payload into frames. The issue becomes relevant when nodeos is started from a snapshot, thus storing all the current deltas into the state log and transferring them when the snapshot block is requested.

In some chains this payload is in the order of multiple gigabytes, making it not only very hard to process since its not streamed in parts and also blocking nodeos processing while its transmitted.

we've tried enabling socket_stream->auto_fragment(true); before socket_stream->async_write(...) but it didn't make any difference https://github.com/EOSIO/eos/blob/177a19faca1ac884856bbd2875fdc0327ef52e64/plugins/state_history_plugin/state_history_plugin.cpp#L140

Ideally nodeos could send the data into multiple messages too if some kind of "more" flag was present to indicate that multiple parts with deltas of the same block would appear. Or more simply implement the fragmentation as per websocket spec. (https://datatracker.ietf.org/doc/html/rfc6455#section-5.4)

The primary purpose of fragmentation is to allow sending a message that is of unknown size when the message is started without having to buffer that message. If messages couldn't be fragmented, then an endpoint would have to buffer the entire message so its length could be counted before the first byte is sent. With fragmentation, a server or intermediary may choose a reasonable size buffer and, when the buffer is full, write a fragment to the network.