mars icon indicating copy to clipboard operation
mars copied to clipboard

Fetch mapper data in advance for shuffle tasks

Open hekaisheng opened this issue 2 years ago • 0 comments

What do these changes do?

There is a barrier now that reducers start to fetch mapper data after all mappers done, this PR try to prefetch map data once one mapper execution is done to break the barrier, it would bring improvement of shuffle performance.

Related issue number

Fixes #xxxx

Check code requirements

  • [ ] tests added / passed (if needed)
  • [ ] Ensure all linting tests pass, see here for how to run them

hekaisheng avatar May 13 '22 09:05 hekaisheng