dangshazi
dangshazi
# Description We maybe encounter a dead lock - netty threads is waiting for lock ## Task stack ``` "HashJoin(joinType=[LeftOuterJoin], where=[(user_id = user_id0)], select=[user_id, user_id0, tag], build=[right]) [Source: HiveSource-smcdrisk_antifraud.th_dp_whitelist_raw_tab] ->...
Avoid specifying "remote-shuffle.manager.rpc-address" when start ShuffleManager in standalone mode
We must specify "remote-shuffle.manager.rpc-address" when start ShuffleManager in standalone mode now. eg. ``` bash /opt/flink-remote-shuffle/active/bin/shufflemanager.sh start remote-shuffle.manager.rpc-address=${IP} ``` The configuration should be removed
Problems ShuffleService can't guarantee resource - storage and network buffers
Monitor *ShuffleManager/ShuffleWorker* CPU usage
Monitor JVM CPU usage
Flink changes `ConnectionID` in 1.15.4. Related issue : [#21358](https://github.com/apache/flink/pull/21358) Error msg: ``` java.lang.NoSuchMethodError: org.apache.flink.runtime.io.network.ConnectionID.(Ljava/net/InetSocketAddress;I)V at com.alibaba.flink.shuffle.plugin.transfer.RemoteShuffleInputGate$FakedRemoteInputChannel.(RemoteShuffleInputGate.java:726) at com.alibaba.flink.shuffle.plugin.transfer.RemoteShuffleInputGate.getChannel(RemoteShuffleInputGate.java:682) at org.apache.flink.runtime.io.network.partition.consumer.InputGate.setChannelStateWriter(InputGate.java:90) at org.apache.flink.runtime.taskmanager.InputGateWithMetrics.setChannelStateWriter(InputGateWithMetrics.java:120) at org.apache.flink.streaming.runtime.tasks.StreamTask.injectChannelStateWriterIntoChannels(StreamTask.java:491) at org.apache.flink.streaming.runtime.tasks.StreamTask.(StreamTask.java:464) at org.apache.flink.streaming.runtime.tasks.StreamTask.(StreamTask.java:363) at org.apache.flink.streaming.runtime.tasks.StreamTask.(StreamTask.java:336)...
TPCDS_Tools.zip无法下载 具体链接：http://dev.hortonworks.com.s3.amazonaws.com/hive-testbench/tpcds/TPCDS_Tools.zip 