mars
mars copied to clipboard
Support get mappers data without index/mapperids
What do these changes do?
This PR removes source_idxes/source_mappers from FetchShuffle to reduce scheduling overhead of shuffle meta and make operands relying on those attributes work on ray task-based shuffle.
The core idea is:
- Remove
source_idxes/source_mappersfromFetchShuffle. - Add
source_idxes/source_mappersto mapper blocks if needed. - Add
chunktocontextwhen executing operand so the operand mapper can get index and add it to mapper blocks.
Related issue number
Fixes #3186
Check code requirements
- [ ] tests added / passed (if needed)
- [ ] Ensure all linting tests pass, see here for how to run them