flink-remote-shuffle icon indicating copy to clipboard operation
flink-remote-shuffle copied to clipboard

Published 20 hours ago • verified publisher icon flink-extended

[FRS-13] Implement ReducePartition for remote shuffle service

Open TanYuxin-tyx opened this issue 2 years ago • 0 comments

What is the purpose of the change

This change introduces the ReducePartition shuffle mode for remote shuffle service.

ReducePartition is a good supplement to the current MapPartition. It has several good features, for example, it can benefit streaming and hybrid shuffle (mixed streaming and batch) and it is suitable for performance in some scenarios.

Brief changelog

  • Support ReducePartition for API/client/transfer/storage/manager layers.
  • Add tests for ReducePartition.

Verifying this change

This change added tests.

TanYuxin-tyx avatar Apr 11 '22 09:04 TanYuxin-tyx