flink-remote-shuffle
flink-remote-shuffle copied to clipboard
flink-extended
Remote Shuffle Service for Flink
When the deployment mode is Kubernetes, can we consider supporting Kubernetes-based high-availability services? In this way, zk is not our only choice. All the HA information relevant for a specific...
### Motivation Web UI is a simple way to improve usability. It can offer some system state and information, like metrics, ShuffleManager/ShuffleWorker information and so on. It can also supply...
### Motivation Based on the rest API, we can remove or add ShuffleWorker dynamically without restarting the cluster. For example, we can remove a bad ShuffleWorker or we can add...
Problems ShuffleService can't guarantee resource - storage and network buffers
This is an umbrella of some further improvements for remote shuffle service. ### Tasks - [x] https://github.com/flink-extended/flink-remote-shuffle/issues/79 - [x] https://github.com/flink-extended/flink-remote-shuffle/issues/81
### Motivation As Flink 1.16 is going to release, the remote shuffle project should also adapt to 1.16. ### Changes - Adapt to Flink 1.16 which means adapting to some...
Flink changes `ConnectionID` in 1.15.4. Related issue : [#21358](https://github.com/apache/flink/pull/21358) Error msg: ``` java.lang.NoSuchMethodError: org.apache.flink.runtime.io.network.ConnectionID.(Ljava/net/InetSocketAddress;I)V at com.alibaba.flink.shuffle.plugin.transfer.RemoteShuffleInputGate$FakedRemoteInputChannel.(RemoteShuffleInputGate.java:726) at com.alibaba.flink.shuffle.plugin.transfer.RemoteShuffleInputGate.getChannel(RemoteShuffleInputGate.java:682) at org.apache.flink.runtime.io.network.partition.consumer.InputGate.setChannelStateWriter(InputGate.java:90) at org.apache.flink.runtime.taskmanager.InputGateWithMetrics.setChannelStateWriter(InputGateWithMetrics.java:120) at org.apache.flink.streaming.runtime.tasks.StreamTask.injectChannelStateWriterIntoChannels(StreamTask.java:491) at org.apache.flink.streaming.runtime.tasks.StreamTask.(StreamTask.java:464) at org.apache.flink.streaming.runtime.tasks.StreamTask.(StreamTask.java:363) at org.apache.flink.streaming.runtime.tasks.StreamTask.(StreamTask.java:336)...
 
flink 1.15.3 master branch AkkaRpcServiceUtils compile fail 
