starrocks icon indicating copy to clipboard operation
starrocks copied to clipboard

[Enhancement]Support batch drop partitions syntax

Open leiyang0324 opened this issue 1 year ago • 1 comments

Why I'm doing:

The current latest version lacks the functionality for batch deletion of partitions. When a large number of partitions need to be deleted, the efficiency is relatively low due to the necessity of obtaining locks during deletion.

What I'm doing:

Add batch deletion syntax and functionality for formal partitions and temporary partitions."

Fixes #issue

What type of PR is this:

  • [ ] BugFix
  • [ ] Feature
  • [x] Enhancement
  • [ ] Refactor
  • [ ] UT
  • [ ] Doc
  • [ ] Tool

Does this PR entail a change in behavior?

  • [x] Yes, this PR will result in a change in behavior.
  • [ ] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • [x] Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • [ ] Parameter changes: default values, similar parameters but with different default values
  • [ ] Policy changes: use new policy to replace old one, functionality automatically enabled
  • [ ] Feature removed
  • [ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • [x] I have added test cases for my bug fix or my new feature
  • [ ] This pr needs user documentation (for new or modified features or behaviors)
    • [ ] I have added documentation for my new feature or new function
  • [ ] This is a backport pr

Bugfix cherry-pick branch check:

  • [x] I have checked the version labels which the pr will be auto-backported to the target branch
    • [ ] 3.2
    • [ ] 3.1
    • [ ] 3.0
    • [ ] 2.5

leiyang0324 avatar Feb 27 '24 07:02 leiyang0324

CLA assistant check
All committers have signed the CLA.

CLAassistant avatar Feb 27 '24 07:02 CLAassistant

@leiyang0324 Excellent job, @gengjun-git and i will help to review this pr, thanks.

nshangyiming avatar Feb 28 '24 11:02 nshangyiming

@leiyang0324 Can you give an example of the new syntax?

gengjun-git avatar Feb 28 '24 11:02 gengjun-git

@gengjun-git Thank you for your reply -- formal partition ALTER TABLE test.test_broke_load_index_vip DROP PARTITIONS IF EXISTS START("2024-02-24") END("2024-02-26") EVERY (INTERVAL 1 DAY); -- temporary partition ALTER TABLE test.test_broke_load_index_vip DROP TEMPORARY PARTITIONS IF EXISTS START("2024-01-01") END("2024-01-05") EVERY (INTERVAL 1 DAY);

leiyang0324 avatar Feb 28 '24 11:02 leiyang0324

@gengjun-git Thank you for your reply -- formal partition ALTER TABLE test.test_broke_load_index_vip DROP PARTITIONS IF EXISTS START("2024-02-24") END("2024-02-26") EVERY (INTERVAL 1 DAY); -- temporary partition ALTER TABLE test.test_broke_load_index_vip DROP TEMPORARY PARTITIONS IF EXISTS START("2024-01-01") END("2024-01-05") EVERY (INTERVAL 1 DAY);

@wangsimo0 @jaogoy PTAL, do you think this syntax is reasonable?

gengjun-git avatar Mar 05 '24 03:03 gengjun-git

@mergify rebase

nshangyiming avatar Mar 25 '24 07:03 nshangyiming

rebase

✅ Branch has been successfully rebased

mergify[bot] avatar Mar 25 '24 07:03 mergify[bot]

Quality Gate Failed Quality Gate failed

Failed conditions
10.3% Duplication on New Code (required ≤ 3%)

See analysis details on SonarCloud

sonarqubecloud[bot] avatar Mar 25 '24 07:03 sonarqubecloud[bot]

[FE Incremental Coverage Report]

:x: fail : 59 / 96 (61.46%)

file detail

path covered_line new_line coverage not_covered_line_detail
:large_blue_circle: com/starrocks/clone/DynamicPartitionScheduler.java 0 2 00.00% [442, 531]
:large_blue_circle: com/starrocks/server/LocalMetastore.java 38 73 52.05% [1585, 1586, 1588, 1589, 1591, 1593, 1594, 1595, 1598, 1599, 1600, 1601, 1602, 1603, 1604, 1605, 1606, 1607, 1610, 1611, 1612, 1614, 1615, 1616, 1618, 1619, 1620, 1621, 1622, 1623, 1624, 1625, 1627, 1647, 1660]
:large_blue_circle: com/starrocks/persist/DropPartitionInfo.java 7 7 100.00% []
:large_blue_circle: com/starrocks/alter/AlterJobMgr.java 1 1 100.00% []
:large_blue_circle: com/starrocks/sql/ast/MultiRangePartitionDesc.java 2 2 100.00% []
:large_blue_circle: com/starrocks/sql/ast/DropPartitionClause.java 11 11 100.00% []

github-actions[bot] avatar Mar 25 '24 09:03 github-actions[bot]

[BE Incremental Coverage Report]

:white_check_mark: pass : 0 / 0 (0%)

github-actions[bot] avatar Mar 25 '24 09:03 github-actions[bot]

@leiyang0324 could you rebase the main branch to resolve conflicts, so that we can run the ut.

nshangyiming avatar Mar 29 '24 08:03 nshangyiming

@leiyang0324 could you rebase the main branch to resolve conflicts, so that we can run the ut.

ok,i will resolve conflicts

leiyang0324 avatar Apr 01 '24 02:04 leiyang0324

image

@leiyang0324 there must be something wrong...

nshangyiming avatar Apr 01 '24 02:04 nshangyiming

image

@leiyang0324 there must be something wrong... I will pull a new branch from the latest master branch, apply the code patch for batch partition deletion to this new branch, and then create a new pull request.

leiyang0324 avatar Apr 01 '24 03:04 leiyang0324

Going to close this pr due to rebase, switch to this new pr: https://github.com/StarRocks/starrocks/pull/43539.

nshangyiming avatar Apr 02 '24 08:04 nshangyiming