Utility to push to an iterable

[dead-claudia]

It'd be very useful to have something that pushes to an iterable, and in fact, it's pretty critical for adapting any push-based API to async iterators.

Of course, there's not just one way to do it, hence why my suggestion takes parameters.

const iterator = Iterator.buffered(writer => {
    // ...
}, options?)

const asyncIterator = AsyncIterator.buffered(writer => {
    // ...
}, options?)

Create a buffered iterator. Up to options.bufferSize entries (minimum and default: 1) may be pushed without a pending next call, and options.retain determines whether to retain the newest ("newest") or oldest ("oldest") entries.

const open = writer.isOpen

Returns true if the writer can accept entries, false otherwise. This changes based on one of four things happening:

• writer.throw being called
• writer.return being called
• iterator.throw/asyncIterator.throw being called
• iterator.return/asyncIterator.return being called

const result = writer.next(value)

Enqueue the value into the iterator/async iterator. Returns false if any value had to be dropped from the buffer, and true otherwise. If throw or return enqueued, a TypeError is thrown.

writer.throw(value)

Enqueue a throw after all next values are consumed. If the writer has already had a throw or return enqueued, a TypeError is thrown.

writer.return(value)

Enqueue a return after all next values are consumed. If the writer has already had a throw or return enqueued, a TypeError is thrown.

[ljharb]

Wouldn't that just be .concat (which is a planned followup to this proposal)? iow, you'd either do [item].values().concat(iter) or iter.concat([item])

[Jamesernator]

Wouldn't that just be .concat (which is a planned followup to this proposal)? iow, you'd either do [item].values().concat(iter) or iter.concat([item])

For sync this is probably the case, although I don't see how this could work for async, in the async case it'd be more like having something like:

import glob from "glob";

const globIterator = AsyncIterator.buffered(writer => {
    const globber = new glob.Glob(someGlobPattern);
    globber.on("match", (entry) => {
        writer.next(entry);
    });
    globber.on("end", () => {
        writer.return();
    });
});

There's no way to do this with .concat as you don't in general have the things you want to concat when it's time to access the iterator.

This pattern is pretty much exactly how you construct ReadableStream or require("node:stream").Readable. Having AsyncIterator.buffered would essentially be a way to create an async iterator directly from a push source.

There is a case that maybe those objects should just be used directly for converting push sources to async iterators, although neither is a JS standard so environments might no in general support them, and both have some overhead due to their both having special support for binary streams.

[devsnek]

regardless of whether this method is justified, this proposal isn't getting any more methods, so it would have to be a followup either way. feel free to continue discussion but i'm gonna mark this closed.