Are updaters the dual of iterators?

[kriskowal]

In the @endo/stream package, I propose the existence of a stream type that emerges from JavaScript async iterators. A stream satisfies the interface of both an async iterator and a stricter parody of an async generator function.

https://github.com/endojs/endo/blob/345ea246ef4f2f8162d5073ed761136cfd7cd2d6/packages/stream/types.d.ts#L10-L20

With this kind of stream, the producer and the consumer are dual stream types. For example, Reader<T> and Writer<T> are both defined in terms of Stream, the difference being what data type flows forward or backward.

https://github.com/endojs/endo/blob/345ea246ef4f2f8162d5073ed761136cfd7cd2d6/packages/stream/types.d.ts#L22-L33

This abstraction allows for the creation of utility methods to bridge streams, passively with makePipe and actively with pump, as implemented.

In the way that Array.from can accept anything that is synchronously iterable, and the proposed Array.fromAsync can lift anything that is either synchronously or asynchronously iterable, these utility methods are able to lift other JavaScript types to streams.

The following diagram illustrates a metaphor for iteration types in JavaScript in terms of sockets and plugs. I used the terms “Async Iterator” and “Stream Reader” interchangeably. I also used “Async Observer” to mean “Stream Writer”. Async generator functions, for example, can be used to make either async iterators or async observers, with some caveats around parity and some TypeScript wrangling for overloads with optional arguments.

The @agoric/notifiers package, which we hope to eventually subsume into Endo, provides an updater type that does not lift easily into this framework. An adapter is necessary for an updater to be accepted in place of a writer, as in the second argument to pump.

I propose that in a future generation of the Notifiers and PubSub APIs, the producer side of either should satisfy the type Writer<T> so that it can participate in this system without an adapter.

[erights]

Now that we're about to make significant changes to the notifier/subscription system, if we're going to replace the Updater type with Writer, now would be a good time to do so. If the change involved only renaming (updateState -> next, finish -> return, fail -> throw) this would be straightforward. I have no problem with next as the new name for updateState though I remain uncomfortable with return and throw as names for non-control-flow signals.

However, this makes a deeper change from Updater to Writer than just a renaming. The current updater.updateState(value) is void return. Whereas writer.next(value) is typed as returning Promise<IterationResult<undefined, undefined>>. Likewise for finish vs return and fail vs throw. Between this and the even/odd priming issue, I am still on the fence about whether and updater-to-writer adapter is warranted.

[erights]

Attn @dtribble @Tartuffo

[kriskowal]

Whether one of these producer-side types should return void hinges on whether they can, should, and do provide a back-pressure signal. Whether they should provide a back-pressure signal depends on what they do instead of retaining an indefinitely growing buffer when the consumer is slower than the producer. I think it’s very reasonable for Updater or Publisher methods to return void if it’s okay to drop messages, or if there’s a suitable reducer to merge the backlog. Publishers can get away with not solving the problem since retaining the entire publication forever is the design, and writing them out should be faster than anything producing updates.

Whether the method names should be next, return, and throw depends entirely on whether it’s desirable to force the developer to use an adapter. For iterators lifting to async iterators, the language’s opinion is that no adapter should be necessary, and that opinion is very useful for writing tests and such, where you can use an array as a fixture for a stream. To be consistent, I don’t think that dropping an asynchronous observer to a synchronous observer should require an adapter either: it is just a matter of the code ignoring the return values.

[kriskowal]

To be clear, the amount of energy I hope we dedicate to this issue is near zero. The worst outcome of not doing this is that someday, somewhere, someone’s going to have to write a very small adapter. The best outcome is that someone (maybe) gets to that point and doesn’t have to write an adapter and feels good about how thoughtful someone else must have been.

[erights]

Both notifier and subscription are multicast channels where the writer must not be vulnerable to the readers, and the readers must not be vulnerable to each other. Even by itself, I think that means backpressure must be absent. Additionally, both are lossy, though in different ways, in order to reduce the implied storage burden on the writer's system.

[kriskowal]

So, given that they’re multicast and lossy, the methods should definitely return void and definitely should not satisfy the Writer<T> interface. That makes them analogous to synchronous observers rather than asynchronous observers. So by way of a syllogism:

1. Asynchronous and synchronous iterators use the names next, return, and throw. The synchronous ones return X and the asynchronous ones return Promise<X>, where X is IteratorResult<T>.
2. Analogously, asynchronous and synchronous observers should use the same method names, but where a sync observer receives X and returns void, the async observer receives X | Promise<X> and also returns void.
3. Generator functions and async generator functions establish a precedent for iterators and observers having the same method names but with different signatures.

I would still conclude that Updaters and Publishers should use the same method names and satisfy the synchronous observer interface. For the purposes of many functions, synchronous observers, asynchronous observers, and stream writers become interchangeable without modification.