feat: multi-service rpc for iroh-one

[fabricedesre]

This patch enables iroh-one rpc control enpoint to serve the same calls as the independent gateway, store and p2p programs. To that end, the following changes are made:

• decouple iroh-one rpc from iroh-gateway rpc.
• create a iroh-one specific grpc definition, with 3 services.
• upddate the rpc macros to support multiple services.

[fabricedesre]

@dignifiedquire I changed or added a bunch of paste!() modifiers (from :lower to :snake, and :camel) to match better the code generated by prost.

I tried to minimize manually written code as much as possible in iroh-one/src/rpc.rs since we just relay calls to the store and p2p services that are already instantiated.

[ramfox]

Hey @fabricedesre! We've been putting in some real thought about how we want to proceed with allowing for communication between an iroh-one process and a iroh cli, and I wanted to gather feedback.

Rather than having a specific RPC API that opens up only the gateway to outside communication, we would rather keep a single RPC API & refactor the RPC servers to allow for passing RPC messages on multiple types of communication channels.

So for the iroh-one case, you would have a gateway, p2p, and store service each with an RPC client that can communicate over mem to each other, and an RPC server that can respond to both mem requests & gRPC requests for interacting with each other & the iroh cli, respectfully.

This would double down on our reliance on the gRPC APIs internally, but would still keep the APIs private.

Question, though: would this setup still suit your use case?

[fabricedesre]

I'm not sure I understood everything you wrote @ramfox :)

The current situation in iroh-one is:

• store and p2p are accessible each over a memory rpc channel.
• the gateway uses these memory channels to coordinate the resolver with store & p2p.
• a grpc (tcp or uds) server exposes the gateway service only. Store & p2p are not accessible from the outside.

What this PR does is:

• no change to the internal store and p2p setup. The gateway still uses memory channels to communicate internally.
• the grpc endpoint, instead of just exposing the Gateway service, also exposes a Store and a P2P service (in gateway_one.proto). When a request is received for one of these services, we either answer directly for the Gateway one, or proxy the call over a memory channel if it's a Store or p2p call.

I'm not sure how it would look like from a client point of view. Each service should be usable independently even if they are all served over the same grpc endpoint, but I don't know if the tonic code generation allows that.

[ramfox]

Ah, okay let me rephrase because I'm really just trying to gather information:

Is it vital for your use case that all outside communication come from one rpc endpoint?

To give you context: We want the iroh cli to be able to be used for both the iroh-one use case and the cloud/microservices use case. There seem to be two options:

1. like this PR, we have, for iroh-one, a new RPC api for the RPC server that combines all three current RPC apis, and proxies any incoming gRPC requests to the internal mem rpc client. There is only a single rpc endpoint to communicate with any of the processes.

• pros: conceptually simpler
• cons: new api to maintain

2. Each RPC server, for gateway, p2p, and store, is refactored to be able to receive communication over mem & grpc. There is a different grpc endpoint for each service.

• pros: still have a single API
• cons: conceptually it is a bit odder

Sidebar: There is a "secret" con to both these approaches (when looked at in the context of the iroh cli). The resolver that does the coordination for commands like add and get is inside the iroh command line binary. This means each rpc call is forced to go through gRPC and we can't leverage the fact that we can use mem transports to get a significant speed up. If you are running an iroh-one process, resolving through the gateway would be significantly faster than calling iroh get.

Internally, we are favoring option 2 because it means there is one fewer API to maintain.

But the iroh cli use case may not be the only use case you were considering when writing this PR, & you favored option 1, so we wanted to check in to see if option 2 would still satisfy your use case.

[fabricedesre]

I favored option 2 because that looked cleaner from the iroh-one perspective, and I envisioned the api to be smart enough to know what to do when being configured with a single grpc access point vs. several ones. I don't know if there's an overhead to run a single grpc endpoint with 3 services vs. 3 endpoints with a single service each.

So... feel free to implement your way, that's not a blocker for me. I feel it's more work though.

[dignifiedquire]

Given the upcoming changes in RPC we won't be merging this as is, so I am going to close this for now. But we are planning to fix the problem this PR addresses, promised :)