mars icon indicating copy to clipboard operation
mars copied to clipboard

Support get mappers data without index/mapperids

Open chaokunyang opened this issue 3 years ago • 0 comments

What do these changes do?

This PR removes source_idxes/source_mappers from FetchShuffle to reduce scheduling overhead of shuffle meta and make operands relying on those attributes work on ray task-based shuffle.

The core idea is:

  • Remove source_idxes/source_mappers from FetchShuffle.
  • Add source_idxes/source_mappers to mapper blocks if needed.
  • Add chunk to context when executing operand so the operand mapper can get index and add it to mapper blocks.

Related issue number

Fixes #3186

Check code requirements

  • [ ] tests added / passed (if needed)
  • [ ] Ensure all linting tests pass, see here for how to run them

chaokunyang avatar Aug 11 '22 13:08 chaokunyang