RPC server for the Narwhal / external consensus API

[huitseeker]

Epic

See MystenLabs/narwhal#44 and spec https://docs.google.com/document/d/1ogieB-Y-PkvINJLfpS9G5Tga4RrmBef3OO0RSHa1fyU/edit#

Description

We would like to stand up an RPC server over IPC to serve the Narwhal/consensus API. We have several endpoints to expose (5 for the validator API, 2 for the block proposer API, 2 for the protocol state, see spec). We want the RPC server to be accessible over IPC only (for security reasons). The chosen initial RPC stack is gRPC.

Eventually, we also want a reference client, which will be used:

• to help customers interface with the server,
• to drive integration tests making repeated calls to the RPC server in known scenarios.

Resources

IPC https://github.com/servo/ipc-channel

We like the Towers ecosystem for modeling request-response patterns:

https://docs.rs/tower/latest/tower/#the-tower-ecosystem https://tokio.rs/blog/2021-05-14-inventing-the-service-trait

One gRPC implementation that follows this pattern: https://github.com/hyperium/tonic Another: https://github.com/google/tarpc

• MystenLabs/narwhal#67
• MystenLabs/narwhal#69
• MystenLabs/narwhal#70
• MystenLabs/sui#5294

[arun-koshy]

I began playing around with tonic last week and was looking into seeing if they have some examples of alternate network channels other than HTTP. I couldn't find anything so I went to check with the discord channel for some advice and this is what I heard back about integrating something like servo/ipc-channel with tonic and this was the response

You may be able to, but gRPC primarily specifies HTTP/2, so most things are pretty focused on HTTP being the protocol. Nonetheless, you can provide your own transport by implementing GrpcService on the client side and then Services for each of the RPCs. You'd also need to translate into and out of the Request constructs used into however you transfer data over IPC. You also may be able to set things up to use IPC as the transport layer, but still speak HTTP/2. All that is to say that it is possible, but it may be a not-insignificant amount of effort and leave you in an area where you won't have many others that will be able to give you guidance.

@bmwill @huitseeker just wanted to check with you both and see if this brings up any red flags with tonic for you guys?

[bmwill]

Is there more context on why we wouldn't be able to use a loopback address?

Also AFAIK gRPC is fundamentally built on top of HTTP/2 so we'd be using http even if we used an ipc channel. IIUC really what this issue is discussing is the ability to replace TCP/IP with IPC for establishing a full-duplex connection between processes.

Tonic is, in my experience, the best option for gRPC and I don't think there are really very many other libraries that come close. The other one listed above, tarpc, isn't a gRPC library but another RPC format IIUC.

[arun-koshy]

This is a very good question and I have no reasons for why we couldn't use a loopback address as it would never hit the network. Maybe this will remove the need for IPC. I will be building out the gRPC server using loopback addr for now. Maybe @huitseeker has some opinions here.

After doing some research on full-duplex connection I think its worth discussing if that is necessary in our case. I am still learning about the endpoints that are being built but from what I understand we are retrieving resources from the mempool that should not require an ongoing connection. When a block (collection) is ready the consensus layer will query for the relevant information. I believe the consensus layer will be polling the mempool for any new block information which means a half-duplex connection could suffice? But maybe it is more efficient to do this polling via an ongoing connection, I am not sure on what cadence blocks (collections) are created.

cc @akichidis who can add additional information on the endpoints themselves.

[huitseeker]

1. I'm ok with a loopback connection initially,
2. The interface between Narwhal and its co-located consensus is definitely bi-directional, but the bulk of the data is going one way (Narwhal to consensus) and the initial implementation doesn't have to be,
3. Folks have been working on IPC compatibility: https://github.com/hyperium/tonic/issues/742