flink-remote-shuffle icon indicating copy to clipboard operation
flink-remote-shuffle copied to clipboard

Published 20 hours ago • verified publisher icon flink-extended

Implement ReducePartition

Open wsry opened this issue 3 years ago • 1 comments

Motivation

As described in the document, ReducePartition is a good supplement of the current MapPartition. It has several good features, for example, it can benefit streaming and hybrid shuffle (mixed streaming and batch) and it is good for performance for some scenarios.

Changes

It is big change including many aspects, including shuffle API enhancement (Flink side), ShuffleMaster change, shuffle write/read change, network protocol change, storage change, fault tolerance change and so on. We need to Implement it step by step.

Tasks

  • [x] https://github.com/flink-extended/flink-remote-shuffle/issues/40

Test

  • UT test.
  • E2E test.
  • Stability test.
  • Test manually on a cluster.

wsry avatar Dec 06 '21 12:12 wsry

@TanYuxin-tyx Is this feature already implemented ?

qiunanx avatar Oct 17 '22 11:10 qiunanx