substantic
Replace capnp with another protocol
While it currently serves reasonably well, Capnp has several drawbacks for Rain:
- The interface and support for some languages is limited (e.g. no RPC for some, Pytohn API is very low-level).
- No support in browser means we need REST(-like) API for monitoring anyway.
- Compiling in some HPC environments may be an additional problem (may not improve with other solutions).
- The message size is limited to 64MB (see #8) which makes large submits (and possibly other information calls) complicated.
- We do not really need the advanced features of Capnp such as remote objects (these also need an additional packet to free even if they have no methods, see https://github.com/capnproto/capnproto/issues/534).
- The current capnp-rust sends every message in 2 packets - a further minor slowdown
- However we do need bi-directional RPC calls, so no simple server/client RPC would do.
Therefore it might be better to use different protocols for different interfaces:
- For client, low latency is not critical while portability and ease of use are, so it may make sense to have a REST API (or gRPC). The monitoring javascript app already uses REST for information.
- It would be nice to leverage serde-json for the messages
- API can be specified with a tool to generate code stubs and verifiers (like Apiary?)
- For server-worker communication, it would be ergonomic to use serde, tarpc or possibly even abomonation with some simple framing.
- For worker-subworker communication, it would be best to have some multi-platform and simple and yet fast protocol. Here the solution is less clear, but the communication schema is simpler (even if ideally with bi-directional messages).
This issue is a basis for discussion of requirements and options. The transition from Capnp is not critical.
For client connection, WebSockets with JSON-RPC 1.0 or a simiar custom protocol (based on json or message pack (MP) or CBOR)seem to be best option:
- WS, JSON and message-pack are easily available (only competitor being Protobufs)
- Unlike JSON, MP and CBOR support binary data and distinguish int/float numbers
- In Rust we can use serde for Json, MP or CBOR, python has simple crates.
- For WS in python there is aiohttp, in Rust probably rust-websocket
- JSON and MP are (surprisingly) reasonably fast to parse (not being the bottleneck at the client side at least for now)
- It would be nice to be able to stream notifications from server to client, as is available in JSON-RPC 1.0 and not in 2.0, but that may not be crucial and we may use MP-RPC or even make our own simple protocol.
This needs to be done before #38.
Partially resolved on v0.3: The governor-executor protocol has been switched to CBOR. The attributes are now transported as JSON even in capnp and remote object retrieval has been simplified, making future migration easier.