mars
mars copied to clipboard
Fetch mapper data in advance for shuffle tasks
What do these changes do?
There is a barrier now that reducers start to fetch mapper data after all mappers done, this PR try to prefetch map data once one mapper execution is done to break the barrier, it would bring improvement of shuffle performance.
Related issue number
Fixes #xxxx
Check code requirements
- [ ] tests added / passed (if needed)
- [ ] Ensure all linting tests pass, see here for how to run them