[Feat] Announce new blocks via IPC

[ljedrz]

Introduction

This is a proposal for a new feature which would allow freshly inserted blocks to be streamed via an IPC socket.

Use cases

The expected users are those who seek to receive information on new blocks as soon as possible (e.g. block explorers), and potentially (if only transmitting further, or with some manual deserialization) without even having to depend on snarkVM. This feature can also be used in test-related tools which produce blocks, e.g. in tandem with the testchain-generator, for immediate consumption by other tools.

Why not implement this in snarkOS?

1. This feature could be used by tools which don't depend on snarkOS, custom nodes, etc.
2. Having this operation in the context of the sequential processing thread guarantees that an aggressively syncing node won't cause the block stream to become unordered.
3. This also ensures that the blocks can be streamed while syncing via any means (CDN, other peers), ensuring maximum stream performance after a potential downtime. This is the ultimate point where block insertion happens (other than directly in the Ledger itself) so no other place requires the announcement code to be attached.
4. In snarkOS, blocks are normally inserted via try_advance_to_next_block, which then notifies other nodes via the BlockLocators mechanism, which means attaching this setup there would incur additional database queries and cloning (as the Block is not readily available).
5. snarkVM already implements the entire persistent storage, so introducing a feature-gated IPC doesn't "break the paradigm" - we're already well into I/O.

Additional considerations

• the blocks could be announced even before their insertion into the ledger - they could be streamed optimistically, with a tiny confirmation message being sent to the IPC once they are fully accepted
• bincode could be changed to some more common serde-compatible serialization format
• the stream currently also receives information on the baked block's height, in order to avoid having to deserialize the block; would any additional information (e.g. the hash) be practical by default?

[kaimast]

Have you thought about exposing a simple high-level function, such as wait_for_next_blocik, instead? That would avoid having to pick a specific communication mechanism for this feature.

the blocks could be announced even before their insertion into the ledger - they could be streamed optimistically, with a tiny confirmation message being sent to the IPC once they are fully accepted

Somewhat of a tangent: For validators, snarkOS sync does not insert blocks into the ledger until they are confirmed by the next block. The goal is for clients to eventually move to the same, safer, mechanism. It would make sense to think about whether we can propagate unconfirmed blocks to reduce latency in the future, and if snarkVM should have a notion of unconfirmed blocks.

[ljedrz]

Have you thought about exposing a simple high-level function, such as wait_for_next_blocik, instead? That would avoid having to pick a specific communication mechanism for this feature.

I'm not sure if this is desirable, as other potential mechanisms (e.g. the network) could be too slow to introduce at this stage, as it could slow down the validator. This way we can ensure that the announcements are near-instantaneous.

[kaimast]

I was thinking of something very simple, like adding a condition variable (or tokio::sync::Notify) to Ledger. There is already a lock around current_block. Whenever current_block is updated, you could call notify` and then have a function like this.

async fn wait_for_block_height(&self, next_height: u32) {
       while self.current_block.read().height() < next_height {
                self.block_notify.notified().await;
       }
}

One could then spawn a task in snarkOS, or whatever uses snarkVM, that waits on this function and executes some custom logic (like writing to a UNIX socket).

[ljedrz]

Part of the design is to not have to have a dependency on snarkVM at all, it's a large library.

[ljedrz]

Rebased, now that the former base is in. It was clean, and there were no other changes.

[ljedrz]

the indexer is not connected to the validator but the feature is enabled

Every time a block is baked, a WARN log is displayed indicating that the feature is active, but the IPC channel is not available.

similarly the indexer goes down mid-stream

The same thing happens, there's just an additional initial ERROR log indicating a socket write error. If the indexer is restarted, the publishing resumes on its own.

would a large block impact consensus at all (serialisation cost)?

I've calculated the average serialization speed in a test run, and got ~857MiB/s (and this was in a backfilling run with parallel serialization, and small blocks, both of which were certain to decrease the calculated speed). This means that a 1MiB block would be serialized in ~1.1ms, and that's with a machine below validator specs.

[kaimast]

I think we should document what functionality the new feature enables somewhere. You could use this README change from the tracing PR as a basis.

You might want to add a note that this is an "unstable" feature and that the wire format can change in the future, just to be safe.

[ljedrz]

I filed a dedicated issue to later document this feature, so as to not cause any conflict with the changes linked by @kaimast.