forte
forte copied to clipboard
Reducer to consume global DataPack
Is your feature request related to a problem? Please describe. Given the pipeline-level data in the pipeline, DataPack now can exist in two places: 1) in the pipeline process stream, and 2) in the pipeline resources (the Global DataPack) shared by all the components. It will be good to have a special processor to handle the Global DataPack.
Describe the solution you'd like Introduce Reducer
- Only Reducer has access to the global DataPack in the pipeline resources.
- Reducer operates on the data collected data in the Global DataPack and calculates the corpus/pipeline-level results.
- To have better extensibility, Reducer is able to process the Global DataPack in different ways by registering different input/output Entries and operators.
Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.
Additional context Add any other context or screenshots about the feature request here.