forte icon indicating copy to clipboard operation
forte copied to clipboard

Reducer to consume global DataPack

Open zhanyuanucb opened this issue 2 years ago • 0 comments

Is your feature request related to a problem? Please describe. Given the pipeline-level data in the pipeline, DataPack now can exist in two places: 1) in the pipeline process stream, and 2) in the pipeline resources (the Global DataPack) shared by all the components. It will be good to have a special processor to handle the Global DataPack.

Describe the solution you'd like Introduce Reducer

  • Only Reducer has access to the global DataPack in the pipeline resources.
  • Reducer operates on the data collected data in the Global DataPack and calculates the corpus/pipeline-level results.
  • To have better extensibility, Reducer is able to process the Global DataPack in different ways by registering different input/output Entries and operators.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

zhanyuanucb avatar Feb 05 '22 01:02 zhanyuanucb