starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[BugFix] Fix the bug of direct schema change

Open trueeyu opened this issue 1 year ago • 2 comments

Why I'm doing:

If the segments of single rowset is overlapping, will should use heap merge when do direct schema change, otherwise the rows is not ordered by sort key.

How to reproduce:

 CREATE TABLE `lineorder_1` (
  `lo_orderkey` int(11) NOT NULL COMMENT "",
  `lo_linenumber` int(11) NOT NULL COMMENT "",
  `lo_custkey` int(11) NOT NULL COMMENT "",
  `lo_partkey` int(11) NOT NULL COMMENT "",
  `lo_suppkey` int(11) NOT NULL COMMENT "",
  `lo_orderdate` int(11) NOT NULL COMMENT "",
  `lo_orderpriority` varchar(16) NOT NULL COMMENT "",
  `lo_shippriority` int(11) NOT NULL COMMENT "",
  `lo_quantity` int(11) NOT NULL COMMENT "",
  `lo_extendedprice` int(11) NOT NULL COMMENT "",
  `lo_ordtotalprice` int(11) NOT NULL COMMENT "",
  `lo_discount` int(11) NOT NULL COMMENT "",
  `lo_revenue` int(11) NOT NULL COMMENT "",
  `lo_supplycost` int(11) NOT NULL COMMENT "",
  `lo_tax` int(11) NOT NULL COMMENT "",
  `lo_commitdate` int(11) NOT NULL COMMENT "",
  `lo_shipmode` varchar(11) NOT NULL COMMENT "",
  INDEX t1 (`lo_partkey`) USING BITMAP COMMENT ''
) ENGINE=OLAP 
DUPLICATE KEY(`lo_orderkey`, `lo_linenumber`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`lo_orderkey`) BUCKETS 1 
PROPERTIES (
"compression" = "LZ4",
"fast_schema_evolution" = "true",
"replicated_storage" = "true",
"replication_num" = "1"
);

insert into lineorder_1 select * from lineorder limit 2000000;

select * from lineorder_1 where lo_orderkey>5427812 and lo_orderkey<=5927812;
select lo_orderkey from lineorder_1 where lo_orderkey>5427812 and lo_orderkey<=5927812;
 
The num of return rows is inconsistent.

What I'm doing:

Use heap merge for direct schema change if the segments of rwoset is overlapping.

What type of PR is this:

  • [x] BugFix
  • [ ] Feature
  • [ ] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Does this PR entail a change in behavior?

  • [ ] Yes, this PR will result in a change in behavior.
  • [x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • [ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • [ ] Parameter changes: default values, similar parameters but with different default values
  • [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
  • [ ] Feature removed
  • [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • [ ] I have added test cases for my bug fix or my new feature
  • [ ] This pr needs user documentation (for new or modified features or behaviors)
    • [ ] I have added documentation for my new feature or new function
  • [ ] This is a backport pr

Bugfix cherry-pick branch check:

  • [x] I have checked the version labels which the pr will be auto-backported to the target branch
    • [x] 3.3
    • [x] 3.2
    • [x] 3.1
    • [x] 3.0
    • [x] 2.5

trueeyu avatar Apr 28 '24 02:04 trueeyu

[FE Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

github-actions[bot] avatar Apr 28 '24 05:04 github-actions[bot]

[BE Incremental Coverage Report]

:white_check_mark: pass : 99 / 114 (86.84%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: be/src/testutil/schema_test_helper.cpp 35 50 70.00% [18, 20, 22, 23, 24, 26, 27, 28, 29, 30, 31, 32, 33, 34, 38]
:large_blue_circle: be/src/storage/schema_change.cpp 9 9 100.00% []
:large_blue_circle: be/src/testutil/tablet_test_helper.cpp 54 54 100.00% []
:large_blue_circle: be/src/storage/task/engine_alter_tablet_task.cpp 1 1 100.00% []

github-actions[bot] avatar Apr 28 '24 05:04 github-actions[bot]

@Mergifyio backport branch-3.3

github-actions[bot] avatar Apr 29 '24 02:04 github-actions[bot]

@Mergifyio backport branch-3.2

github-actions[bot] avatar Apr 29 '24 02:04 github-actions[bot]

@Mergifyio backport branch-3.1

github-actions[bot] avatar Apr 29 '24 02:04 github-actions[bot]

@Mergifyio backport branch-3.0

github-actions[bot] avatar Apr 29 '24 02:04 github-actions[bot]

@Mergifyio backport branch-2.5

github-actions[bot] avatar Apr 29 '24 02:04 github-actions[bot]

backport branch-3.3

✅ Backports have been created

mergify[bot] avatar Apr 29 '24 02:04 mergify[bot]

backport branch-3.2

✅ Backports have been created

mergify[bot] avatar Apr 29 '24 02:04 mergify[bot]

backport branch-3.1

✅ Backports have been created

mergify[bot] avatar Apr 29 '24 02:04 mergify[bot]

backport branch-3.0

✅ Backports have been created

mergify[bot] avatar Apr 29 '24 02:04 mergify[bot]

backport branch-2.5

✅ Backports have been created

mergify[bot] avatar Apr 29 '24 02:04 mergify[bot]