Ray-based Shuffle For Mars

[chaokunyang]

This issue is used for tracking Ray-based Shuffle For Mars progress:

• [x] Ray Future-based Shuffle MEP：https://github.com/mars-project/meps/pull/2
• [x] Shuffle Meta optimization #3055
• [x] Ray Future-based Shuffle
  • [x] Basic Ray Shuffle Implementation #3040
  • [x] Shuffle object GC
  • [x] Shuffle operands compatibility
• [ ] Push-based Ray Shuffle
  • [ ] Push-based ray shuffle MEP
  • [ ] Scheduling all ray tasks in placement group
  • [ ] Pipelined push-based Shuffle
    • [ ] Push strategy optimization. Skip push shuffle blocks whose size exceed specified size.
    • [ ] Support configure merge factor.
    • [ ] Support configure concurrent running mapper subtasks.
    • [ ] Support adjust merge factor adaptively
    • [ ] Support adjust concurrent running mapper subtasks adaptively to reduce write amplification
• [ ] Handle shuffle slow sutbask by speculative execution
• [ ] Dynamic merge shuffle partitions
• [ ] Dynamic split shuffle partitions
  • [ ] Dynamic detect skewed partitions
  • [ ] Dynamic optimize skewed join
  • [ ] Support sub chunks [ ] Shuffle Failover based lineage reconstruction

[wjsi]

I've implemented a half-done branch https://github.com/wjsi/mars/tree/feature/shuffle_svc for shuffle. The reason why it is not merged is that currently storage service lacks the capability of remote readers and writers. Now working on it.

[chaokunyang]

https://github.com/wjsi/mars/tree/feature/shuffle_svc

Is there a design doc for this?

[wjsi]

https://github.com/wjsi/mars/tree/feature/shuffle_svc

Is there a design doc for this?

Not yet. I'll write it soon.