hadron
hadron copied to clipboard
Transactional Pipeline to Stream outputs
Transactional Pipeline -> Stream outputs.
Introduce the new streamOutput field for pipeline stages.
- Stream outputs will be required to be supplied as part of the pipeline
ackpayload, just as a stage output is required. - The payloads will be written to disk along side the pipeline stage's normal output, but will be nested under a separate prefix as
{prefix}{offset}{stage_name}. - A notification will prompt the Stream controller that Pipeline Stream outputs need to be added to the Stream. It will transactionally copy the data to the Stream in proper order, preserving Stream controller encapsulation.
- This will provide a more structured model for transactional processing overall.
- Only a normal stage output can be used as input to another stage, not stream outputs.