spark icon indicating copy to clipboard operation
spark copied to clipboard

[SPARK-39950][SQL] It's unnecessary to materialize BroadcastQueryStage firstly, because the BroadcastQueryStage does not timeout in AQE.

Open weixiuli opened this issue 3 years ago • 4 comments

What changes were proposed in this pull request?

In the https://issues.apache.org/jira/browse/SPARK-33933, it materializes BroadcastQueryStage firstly to try to avoid broadcast timeout in AQE, but the BroadcastQueryStage does not timeout in AQE any more, so we should not materialize BroadcastQueryStage firstly.

Why are the changes needed?

It's unnecessary to materialize BroadcastQueryStage firstly in AQE.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass CI.

weixiuli avatar Aug 01 '22 14:08 weixiuli

Can one of the admins verify this patch?

AmplabJenkins avatar Aug 01 '22 20:08 AmplabJenkins

but the BroadcastQueryStage does not timeout in AQE any more

@weixiuli Could you point out the timeout issue no longer exists after which pr in the pr description?

cc @cloud-fan @viirya

LuciferYang avatar Aug 04 '22 04:08 LuciferYang

Additionally, does this pr just revert SPARK-33933?

LuciferYang avatar Aug 04 '22 05:08 LuciferYang

looks fine, but let's think a bit more and see if there is any benefit to submit broadcast jobs first. cc @yaooqinn @maryannxue

cloud-fan avatar Aug 05 '22 13:08 cloud-fan

@weixiuli Could you point out the timeout issue no longer exists after which pr in the pr description? cc @cloud-fan @viirya

@weixiuli @cloud-fan Hey, same questions with @LuciferYang , could you please point out which pr completely fix this issue?

boneanxs avatar Nov 08 '22 03:11 boneanxs

here you are: https://github.com/apache/spark/commit/0c94e47aecab0a8c346e1a004686d1496a9f2b07

cloud-fan avatar Nov 08 '22 07:11 cloud-fan

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

github-actions[bot] avatar Feb 17 '23 00:02 github-actions[bot]