flink-remote-shuffle
flink-remote-shuffle copied to clipboard
flink-extended
Remote Shuffle Service for Flink
# Description We maybe encounter a dead lock - netty threads is waiting for lock ## Task stack ``` "HashJoin(joinType=[LeftOuterJoin], where=[(user_id = user_id0)], select=[user_id, user_id0, tag], build=[right]) [Source: HiveSource-smcdrisk_antifraud.th_dp_whitelist_raw_tab] ->...
## What is the purpose of the change This change introduces the `ReducePartition` shuffle mode for remote shuffle service. `ReducePartition` is a good supplement to the current `MapPartition`. It has...
### Motivation v1beta1 support has been dropped since 1.22. ### Changes Use apiextensions v1 instead of v1beta1 for remote shuffle operator crd. ### Test Test manually on a cluster.
In our usage, we encounter a case where the shuffle worker registers timeout and triggers a fatal error, but the shuffle worker process does not exit and this leads to...
This is the first part to introduce the shuffle worker deployed by stateful set. It will complete with three PR as plan: - Introduce the statefulset builder and parameters (this...
## What is the purpose of the change SortBuffer can improve read performance significantly, but it doesn't support read data from a specific channel. The development of some new functions,...
By default, the shuffleManagerClient will print the log as below. But the worker Instance id is not human readable. It can not quick link to the target ShuffleWorker. We need...
In the current kubernetes mode, the rss use the hostPath or the emptyDir to store the shuffle data. But, In our company, the local disk resource is very rare and...
Avoid specifying "remote-shuffle.manager.rpc-address" when start ShuffleManager in standalone mode
We must specify "remote-shuffle.manager.rpc-address" when start ShuffleManager in standalone mode now. eg. ``` bash /opt/flink-remote-shuffle/active/bin/shufflemanager.sh start remote-shuffle.manager.rpc-address=${IP} ``` The configuration should be removed
I think it's important to support K8S high availability, and I don't see any progress in the community. The FSR-9 seems to be dead, so I expect it to be...
