codeflare icon indicating copy to clipboard operation
codeflare copied to clipboard

Published 20 hours ago • verified publisher icon project-codeflare

Data splitter

Open raghukiran1224 opened this issue 4 years ago • 1 comments

Overview

As a CFP user, I would like to split a dataset (e.g., np array, pandas dataframe) into smaller objects that can then be fed into other nodes/pipeline. This is especially useful when we have compute intensive tasks and would like to parallelize it easily.

Acceptance Criteria

  • [x] Design for splitter, should be simple and intuitive
  • [ ] Implementation as an extension to the Node construct
  • [x] Tests

Questions

  • What type of semantics does the splitter node define?

Assumptions

Reference

raghukiran1224 avatar Jul 22 '21 13:07 raghukiran1224

The basic utility has been added, putting it as an actual node needs more work.

raghukiran1224 avatar Jul 30 '21 20:07 raghukiran1224