wankun
As a followup of SPARK-38965, if there are no BLOCK_APPEND_COLLISION exception, shuffle fetcher and shuffle pusher do not handle the exception returned from server, and RetryingBlockTransferor will retry if there...
Hi, @otterc Could you help to review this PR ? thanks
> @wankunde > > > Send finalize RPCs will block the main thread due to creating connection to some unreachable nodes. > > Which main thread are you referring to...
Hi, @otterc could you help me to review this PR? Or should I provide more information on this issue ?
> So the issue is that the wait period timer doesn't take into account the time for connection creation which is a bug. However, in this PR you are adding...
Hi, @otterc @mridulm , I updated the code, could you help to review the new code?
Hi, @mridulm @otterc New processing flow: * Send all the finalize RPC tasks and wait for merged status in a new thread pool. * Wait for `shuffleMergeResultsTimeoutSec` in main thread...
I'm sorry for the late reply, I have updated the code.
> Can we add a test? or at least can you describe how you tested it? A simple UT ``` test("join with too many duplicate key") { withSQLConf(SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1")...
@zhengruifeng I'm sorry, I made a mistake, it should be `val duplicateKeyNumber = 2L + Integer.MAX_VALUE`