rpc over websockets

[zenhack]

I was looking at the Sandstorm app-index's implementation, and rather than using capnp rpc for submissions it uses an ad-hoc protocol, with a comment indicating that this is blocked on capnp-rpc over the internet.

I'd like to clean this up by just speaking capnp rpc over a websocket, which seems like it ought to be straightforward enough. The first step would be an implementation of AsyncIoStream on top of WebSocket, which probably belongs in kj.

There's a slight impedence mismatch here, since websockets is message oriented while AsyncIoStream is a streaming interface, but for capnproto's purposes it's fine for the API to obscure this detail.

[kentonv]

Hmm, I think it's pretty important that capnp-over-WebSocket actually respects the WebSocket framing, i.e. encoding one capnp message in one WebSocket message. So, it might be necessary to implement an alternative VatNetwork that uses WebSocket rather than AsyncIoStream...

[zenhack]

Sounds reasonable to me; changed the title accordingly.

But it seems like most of the VatNetwork implementation would be the same as for TwoPartyVatNetwork; perhaps instead we could just factor out the streaming vs. datagram difference, and add a new constructor for this case (so we'd have one that takes AsyncIoStream, one that takes AsyncCapabilityStream, and then some new interface (unless there is already something suitable) that is datagram-oriented).

[kentonv]

That makes sense to me.

Probably the right abstraction is basically to virtualize the serialize-async.h interfaces, like:

class MessageSocket {
public:
  virtual kj::Promise<MessageReaderAndFds> readMessage(
      kj::ArrayPtr<kj::AutoCloseFd> fdSpace, ReaderOptions options = ReaderOptions()) = 0;
  virtual kj::Promise<void> writeMessage(kj::ArrayPtr<const int> fds, MessageBuilder& builder) = 0;
};

[JonasVautherin]

@zenhack: you did resolve this with https://github.com/capnproto/capnproto/pull/1204, and hence it can be closed, right?

I ended up here because I was curious about why WebSockets are using MessageStream instead of AsyncIoStream directly, and Kenton said (above):

Hmm, I think it's pretty important that capnp-over-WebSocket actually respects the WebSocket framing, i.e. encoding one capnp message in one WebSocket message.

May I ask why? WebSockets go over TCP, which is suitable for AsyncIoStream. Also, WebSockets just feel "more constrained" than TCP (WebSocket has to send the whole message, whereas TCP can send more or less bytes as it desires), which would tempt me into thinking that "if it works with TCP, it should work with WebSockets".

What makes it important to respect the WebSocket framing?

[kentonv]

@JonasVautherin Sure, we could use a "byte-stream-over-WebSocket" design, where we implement AsyncIoStream on top of WebSocket, wrapping bytes in meaningless frames on the way out, and discarding the frames on the way back in. There would be no relationship between WebSocket frame boundaries and Cap'n Proto message boundaries -- one frame could contain multiple messages, a fragment of a message, etc.

This would "work", but it would be annoying: On the sending end, most implementations would probably just write one frame per message anyway (because that's the easiest thing to do), but on the receiving end, you wouldn't be able to rely on that, and you'd have to write ugly code to reassemble messages from fragments. If we simply say that the sender is required to send exactly one message per frame, then the receiving code gets simpler.

[zenhack]

Yes, we can close this.