async-stream icon indicating copy to clipboard operation
async-stream copied to clipboard

what about when you *need* a parallel operation

Open dominictarr opened this issue 11 years ago • 4 comments

say, a merge stream, that takes N sorted streams and merges them into one stream? (it would pull from each stream, but output chunks in correct order - for example, to do a merge sort, or LSMT compaction)

A simpler one would just to merge multiple streams into one, but not control the order.

dominictarr avatar Feb 10 '14 01:02 dominictarr

I suppose that should need to be written. Take a look at co-cat which is useful to concatenate strings and could be used as a guide to write a module that does what you are asking for.

mciparelli avatar Feb 10 '14 02:02 mciparelli

co supports yielding arrays for executing in parallel, so something like this should work:

var chunks = yield [
  streamA(end),
  streamB(end),
  streamC(end)
]

juliangruber avatar Feb 10 '14 07:02 juliangruber

I didn't know that! hmm, I suspect you need to use straight callbacks in this case, because what if streamA calls back 3x as fast as streamB which calls back 3x as fast as streamC. you want to get 9 out of streamA for each item from streamC.

dominictarr avatar Feb 10 '14 10:02 dominictarr

Aaah didn't read you wanted to get multiple chunks per stream.

Something like this:?

var cat = require('co-cat');

var chunks = yield cat([
  read(n, streamA),
  read(n, streamB),
  read(n, streamC)
]);
function read(n, fn){
  var todo = +n;
  return function*(end){
    var data = yield fn(end);
    if (!end && data && todo--) return data;
  }
}

juliangruber avatar Feb 10 '14 11:02 juliangruber