How to spread records in a pipeline to multiple readers

[obones]

Node.js Version

v18.20.4

NPM Version

v10.7.0

Operating System

Windows/Linux

Subsystem

stream

Description

Hello,

I currently have the following simple stream pipeline:

source --> transform --> destination

Quite expectedly, the transform stream processes one item at a time from the source and this does what I need it to do. The method given to transform is an async one as it needs to await a call to an HTTPs API which will take longer than the time it takes to produce the next record. In an effort to improve processing time, I would like to split the record processing across multiple consumers like so:

         /--  reader 1  --\
        /---  reader 2  ---\
source --       ...        -- destination
        \--- reader N-1 ---/
         \--  reader N  --/

I have read question #2707 but this describes a case where the same record exiting a stream is duplicated across multiple downward streams but I don't want duplication, just somewhat "parallel" execution.

Is this something that is possible within the stream ecosystem? If yes, how would you suggest that I write it?

Thanks for your help

Minimal Reproduction

No response

Output

No response

Before You Submit

• [X] I have looked for issues that already exist before submitting this
• [X] My issue follows the guidelines in the README file, and follows the 'How to ask a good question' guide at https://stackoverflow.com/help/how-to-ask