starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Enhancement] pushdown or predicate (4): support parse and evaluate or predicate

Open ZiheLiu opened this issue 10 months ago • 3 comments

Why I'm doing:

Parse OR predicate

Parse OR expr to predciate tree.

Evaluate

Before:

  • Iterate over each ColumnPredicate in unordered_map<ColumnId, vector<ColumnPredicate*>>, where the relationship between ColumnPredicates is AND.

After:

  • Iterate over each ColumnPredicate in PredicateTree, where the relationship between ColumnPredicates is AND or OR.

What I'm doing:

Parse OR preidcate

If Type is OR, to convert OR to AND to utilize existing parsing logic, perform the Inverted operation in normalize_xxx_predicate and build_olap_filters. For example, pred_c1_1 or pred_c1_2 or pred_c2_1 or pred_c2_2 will be converted to !(!pred_c1_1 and !pred_c1_2) or !(!pred_c2_1 and !pred_c2_2)

The specific steps are as follows:

  1. When normalizing predicates to value ranges, pass true to <Inverted> of each normalize_xxx_predicate.
  2. When building olap filters by the normalized value ranges, pass true to <Inverted> of ColumnValueRange::to_olap_filter.

Evaluate

Introduce three methods evaluate, evaluate_and, evaluate_or for each type of PredicateTreeNode. Specially, the evaluate method of PredicateTreeAndNode will use evaluate_branchless for non_vectorized ColumnPredicate.

Relates to #43801.

What type of PR is this:

  • [ ] BugFix
  • [ ] Feature
  • [x] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Does this PR entail a change in behavior?

  • [ ] Yes, this PR will result in a change in behavior.
  • [x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • [ ] Parameter changes: default values, similar parameters but with different default values
  • [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
  • [ ] Feature removed
  • [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • [x] I have added test cases for my bug fix or my new feature
  • [ ] This pr needs user documentation (for new or modified features or behaviors)
    • [ ] I have added documentation for my new feature or new function
  • [ ] This is a backport pr

Bugfix cherry-pick branch check:

  • [x] I have checked the version labels which the pr will be auto-backported to the target branch
    • [ ] 3.3
    • [ ] 3.2
    • [ ] 3.1
    • [ ] 3.0
    • [ ] 2.5

ZiheLiu avatar Apr 23 '24 06:04 ZiheLiu

[FE Incremental Coverage Report]

:white_check_mark: pass : 7 / 7 (100.00%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: com/starrocks/qe/scheduler/TFragmentInstanceFactory.java 3 3 100.00% []
:large_blue_circle: com/starrocks/qe/SessionVariable.java 4 4 100.00% []

github-actions[bot] avatar Apr 28 '24 05:04 github-actions[bot]

[BE Incremental Coverage Report]

:white_check_mark: pass : 370 / 452 (81.86%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: be/src/connector/lake_connector.cpp 0 30 00.00% [78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 91, 92, 95, 134, 138, 245, 249, 340, 343, 345, 347, 348, 350, 365, 366, 544, 545, 546]
:large_blue_circle: be/src/exec/olap_scan_node.cpp 0 15 00.00% [537, 538, 539, 540, 541, 542, 543, 544, 545, 546, 548, 549, 551, 671, 673]
:large_blue_circle: be/src/exec/tablet_scanner.cpp 0 4 00.00% [73, 152, 282, 286]
:large_blue_circle: be/src/exec/olap_utils.h 4 8 50.00% [123, 124, 125, 127]
:large_blue_circle: be/src/exec/pipeline/scan/olap_chunk_source.cpp 7 11 63.64% [449, 451, 623, 624]
:large_blue_circle: be/src/storage/predicate_tree/predicate_tree.cpp 10 12 83.33% [87, 88]
:large_blue_circle: be/src/storage/predicate_tree/predicate_tree.hpp 80 94 85.11% [166, 167, 251, 254, 255, 258, 266, 295, 296, 324, 344, 347, 348, 351]
:large_blue_circle: be/src/exec/olap_scan_prepare.cpp 200 209 95.69% [224, 242, 245, 309, 457, 991, 992, 994, 995]
:large_blue_circle: be/src/storage/predicate_tree/predicate_tree.h 1 1 100.00% []
:large_blue_circle: be/src/exec/olap_scan_prepare.h 1 1 100.00% []
:large_blue_circle: be/src/exec/pipeline/scan/olap_scan_context.cpp 17 17 100.00% []
:large_blue_circle: be/src/exec/pipeline/fragment_executor.cpp 3 3 100.00% []
:large_blue_circle: be/src/exec/pipeline/scan/olap_scan_context.h 1 1 100.00% []
:large_blue_circle: be/src/exprs/expr.cpp 1 1 100.00% []
:large_blue_circle: be/src/exec/olap_common.cpp 12 12 100.00% []
:large_blue_circle: be/src/storage/rowset/segment_iterator.cpp 33 33 100.00% []

github-actions[bot] avatar Apr 28 '24 06:04 github-actions[bot]