starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Feature] Support Skew join in FE

Open before-Sunrise opened this issue 1 year ago • 4 comments

Why I'm doing:

skew values use boradcast join, and other values use shuffle join for skew shuffle join

What I'm doing:

support skew join optimization in FE. This is the first pr of https://github.com/StarRocks/starrocks/pull/48655

Fixes #issue

What type of PR is this:

  • [ ] BugFix
  • [x] Feature
  • [ ] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Does this PR entail a change in behavior?

  • [ ] Yes, this PR will result in a change in behavior.
  • [x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • [ ] Parameter changes: default values, similar parameters but with different default values
  • [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
  • [ ] Feature removed
  • [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • [x] I have added test cases for my bug fix or my new feature
  • [ ] This pr needs user documentation (for new or modified features or behaviors)
  • [ ] I have added documentation for my new feature or new function
  • [ ] This is a backport pr

Bugfix cherry-pick branch check:

  • [x] I have checked the version labels which the pr will be auto-backported to the target branch
    • [x] 3.3
    • [ ] 3.2
    • [ ] 3.1
    • [ ] 3.0
    • [ ] 2.5

before-Sunrise avatar Aug 21 '24 07:08 before-Sunrise

Quality Gate Failed Quality Gate failed

Failed conditions
3.5% Duplication on New Code (required ≤ 3%)

See analysis details on SonarCloud

sonarqubecloud[bot] avatar Aug 27 '24 02:08 sonarqubecloud[bot]

Quality Gate Failed Quality Gate failed

Failed conditions
3.9% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

sonarqubecloud[bot] avatar Mar 13 '25 07:03 sonarqubecloud[bot]

[Java-Extensions Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

github-actions[bot] avatar Mar 13 '25 09:03 github-actions[bot]

[FE Incremental Coverage Report]

:white_check_mark: pass : 495 / 560 (88.39%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: com/starrocks/common/util/ProfilingExecPlan.java 2 9 22.22% [244, 245, 246, 247, 248, 249, 251]
:large_blue_circle: com/starrocks/sql/optimizer/operator/physical/PhysicalSplitProduceOperator.java 6 14 42.86% [49, 50, 53, 54, 57, 58, 63, 68]
:large_blue_circle: com/starrocks/sql/optimizer/operator/physical/PhysicalConcatenateOperator.java 7 14 50.00% [41, 53, 54, 57, 58, 61, 62]
:large_blue_circle: com/starrocks/sql/optimizer/operator/physical/PhysicalSplitConsumeOperator.java 11 19 57.89% [65, 66, 69, 70, 73, 74, 79, 84]
:large_blue_circle: com/starrocks/planner/SplitCastPlanFragment.java 29 40 72.50% [63, 85, 86, 91, 92, 96, 97, 101, 102, 106, 107]
:large_blue_circle: com/starrocks/planner/SplitCastDataSink.java 29 34 85.29% [80, 81, 86, 87, 104]
:large_blue_circle: com/starrocks/sql/optimizer/operator/Projection.java 12 14 85.71% [185, 186]
:large_blue_circle: com/starrocks/sql/optimizer/rule/tree/SkewShuffleJoinEliminationRule.java 193 206 93.69% [147, 220, 221, 222, 322, 369, 370, 379, 380, 381, 400, 430, 445]
:large_blue_circle: com/starrocks/qe/scheduler/dag/ExecutionDAG.java 24 25 96.00% [575]
:large_blue_circle: com/starrocks/sql/plan/PlanFragmentBuilder.java 117 120 97.50% [3962, 4078, 4079]
:large_blue_circle: com/starrocks/qe/SessionVariable.java 8 8 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/QueryOptimizer.java 7 7 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/operator/physical/PhysicalCTEAnchorOperator.java 4 4 100.00% []
:large_blue_circle: com/starrocks/planner/DataPartition.java 1 1 100.00% []
:large_blue_circle: com/starrocks/common/LocalExchangerType.java 8 8 100.00% []
:large_blue_circle: com/starrocks/sql/plan/ExecPlan.java 2 2 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/operator/physical/PhysicalHashJoinOperator.java 4 4 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/LogicalPlanPrinter.java 8 8 100.00% []
:large_blue_circle: com/starrocks/analysis/SlotDescriptor.java 2 2 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/operator/OperatorType.java 3 3 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/rule/implementation/HashJoinImplementationRule.java 3 3 100.00% []
:large_blue_circle: com/starrocks/planner/UnionNode.java 8 8 100.00% []
:large_blue_circle: com/starrocks/qe/feedback/guide/JoinTuningGuide.java 3 3 100.00% []
:large_blue_circle: com/starrocks/common/FeConstants.java 1 1 100.00% []
:large_blue_circle: com/starrocks/sql/optimizer/OptExpressionVisitor.java 3 3 100.00% []

github-actions[bot] avatar Mar 13 '25 09:03 github-actions[bot]

[BE Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

github-actions[bot] avatar Mar 13 '25 09:03 github-actions[bot]