[RFC] Create Easy Actor Interface via WASI and stdio

[autodidaddict]

Summary

This RFC proposes a new bi-directional communication mechanism between WebAssembly guest modules (actors) and the wasmCloud host runtime that involves information exchange across captured/managed stdio pipes.

Rationale

The current wasmbus protocol requires the wasmCloud host to implement eight functions and the guest module to implement 1 and make extensive use of the functions exported by the host. Performing a single operation involves a complicated dance of sending numbers back and forth across the guest/host boundary and manually manipulating and extracting bytes to and from the module's linear memory.

The current protocol is also fairly inefficient. The most costly thing we can do in a WebAssembly host is cross the guest/host boundary, and we do that multiple times per invocation right now in order to maintain statelessness and optimize for reading and writing "just numbers".

While this protocol is platform and language agnostic, and any wasm32-targeting language should be able to implement it, the barrier to entry could be considered too high, with too much friction involved in someone creating a new actor SDK language binding.

The proposition is that instead of the complicated suite of "pointer hinting" functions we have now, a simpler exchange of data could be facilitated just by leveraging the stdio (pipes) capabilities of WASI. This would allow us to read and write entire blobs of data without having to manually manipulate pointers or linear memory (the WASI support in the underlying engine would take care of this for us).

Components and Implementation

There are two main components to the implementation of this. The first would be modifying the host such that it sends invocations to the actors (guest modules) via stdio, and reads data back from the modules via stdio. The following pseudocode describes the processing loops that would exist in the host and in the actors. The processing loop for actors would likely be hidden and simplified by the various language-specific SDKs.

This proposal does not suggest making any changes to the RPC traffic carried across the lattice via NATS. This is only for local communication between a host and actor.

Comm Loop (Host)

The host is responsible for plucking inbound messages off the wire (NATS) destined for a given actor. The invocation data is then sent to the actor and a loop is then started. This loop reads data from the actor's stdout until the data is a invocation response. Invocations pulled from stdout are processed by the host (e.g. by sending to the appropriate capability provider)

inv = deserialize(msg) // e.g. from NATS
stdio.write inv(msg)
do
   resp = stdio.read
   if resp is inv
      process(resp)
until resp is inv-r

Comm Loop (Guest)

The guest communications loop involves an infinite loop of waiting for new data to come in on stdin. Once new data (an invocation) does arrive, then the guest does whatever work is necessary. If the guest needs to make host calls (send invocations to actors or providers), then the guest will write an invocation to stdout and get an invocation response in return.

When the guest is finally finished processing the instigating invocation, it will return an invocation response to the host, signifying that it's done with its work.

while msg = stdio.read do
    <do work>
    stdio.write inv
    result = stdio.read inv-r
    stdio.write inv
    result = stdio.read inv-r
    <do work>
    stdio.write inv-r
end

Sample Exchange - Key-Value Counter

The following describes (roughly) the stdio message exchange pattern as it would be implemented in one of our more commonly used examples - the key value counter.

sequenceDiagram
    Host->>+Guest: Inv (HttpServer.HandleRequest)
    Guest->>+Host: Inv (KeyValue.Get)
    Host->>-Guest: Inv-R
    Guest->>+Host: Inv (KeyValue.Set)
    Host->>-Guest: Inv-R
    Guest->>-Host: Inv-R

Message Exchange Protocol

The reasoning behind this particular protocol is we want to make it simple and easy to read, generate, and parse. We do not want to require anyone to use things like protobufs or any other heavyweight format for exchanging data (remember that the format of the raw payload remains opaque to the host, and can be msgpack or whatever is dictated by the contract).

The other main requirement is since we're using stdio, we need to ensure that we don't accidentally treat some token in the middle of encoded data as a newline. This is why we use a preamble line that contains the payload size, allowing us to read a stream of bytes without applying meaning to newlines.

Transmitting an Invocation

While we have an Invocation struct already used for the NATS-based/lattice RPC transmission, we don't actually need that structure for this use case. For example, we don't need the hash, claims, anti-forgery token, etc. Instead we can use a 2-line stanza optimized for stdio where we can either read to the end of a line or read a fixed number of bytes from the buffer.

To tell the other side of a call that we're sending an invocation, we can use the following format:

INV [target] [contract] [link] <operation> <payload_size>\n
<raw binary payload>

The target, contract, and link fields are all optional and only needed when sending an invocation destined for a provider. When sending to an actor, the target field can be either the public key or the call alias (the same way it works today). The important mandatory fields from the first line are the operation and payload_size.

Transmitting an Invocation Response

Sending an invocation response to the other side of this exchange involves the following stanza. Here we send a numeric result code (0 for failure, 1 for success, the same as we do today). If the code indicates failure, the binary payload is assumed to be a UTF-8 encoded string containing a description of the error. If the code indicates success, then the payload is treated as the opaque blob it should be and is delivered accordingly.

INV-R <result_code> <payload_size>\n
<raw binary payload>

[stevelr]

How do wasm runtimes support WASI api libraries without incurring the memory mangling overhead?

Are pipes a standard supported by multiple wasm runtimes?

[autodidaddict]

You can use pipes to talk to wasi stdio from wasm3, wasmer, or wasmtime.

[brooksmtownsend]

Can we specify this protocol using wit and https://github.com/bytecodealliance/wit-bindgen? That way we can support running with guest languages out of the box, and then add codegen support when we can place it on our roadmap

[autodidaddict]

this protocol doesn't need to be specified with anything, let alone wit. We're sending opaque blobs here, so we don't actually need code generation for this protocol. Further, the preamble line is just ASCII text, further eliminating the need for overhead like wit.

WASI gets us running with guest languages out of the box, not wit or wit-bindgen.

[brooksmtownsend]

WASI gets us running with guest languages out of the box, not wit or wit-bindgen.

Okay great, I just wanted to have the simplest path forward to support additional languages. I'm mostly interested in adopting community Wasm standards wherever it makes sense and I think getting the little helper functions to parse the preamble line written in various languages is probably the only step we'd need

[autodidaddict]

Parsing the preamble line is basically just splitting on spaces/whitespace. At least for now, it feels like using wit-bindgen for that would be using a sledgehammer.

[brooksmtownsend]

@autodidaddict this work has been supplanted by Wasm components right? Any reservation to closing this issue?

[autodidaddict]

Correct this can close