enhance: Refactor DC comapction trigger for betther scalability

[XuanYang-cn]

Policy will be easier to write. Trigger will be easier to maintain.

pr: #35961 See also: #35584

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: XuanYang-cn To complete the pull request process, please assign jiaoew1991 after the PR has been reviewed. You can assign the PR to them by writing /assign @jiaoew1991 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• internal/datacoord/OWNERS

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

[mergify[bot]]

@XuanYang-cn Please associate the related pr of master to the body of your Pull Request. (eg. “pr: #”)

[mergify[bot]]

@XuanYang-cn

Invalid PR Title Format Detected

Your PR submission does not adhere to our required standards. To ensure clarity and consistency, please meet the following criteria:

1. Title Format: The PR title must begin with one of these prefixes:

• feat: for introducing a new feature.
• fix: for bug fixes.
• enhance: for improvements to existing functionality.
• test: for add tests to existing functionality.
• doc: for modifying documentation.
• auto: for the pull request from bot.

2. Description Requirement: The PR must include a non-empty description, detailing the changes and their impact.

Required Title Structure:

[Type]: [Description of the PR]

Where Type is one of feat, fix, enhance, test or doc.

Example:

enhance: improve search performance significantly 

Please review and update your PR to comply with these guidelines.

[mergify[bot]]

@XuanYang-cn E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 86.17886% with 51 lines in your changes missing coverage. Please review.

Project coverage is 81.09%. Comparing base (d3bf7a2) to head (ac1fdaf). Report is 148 commits behind head on 2.4.

Files with missing lines	Patch %	Lines
internal/datacoord/compaction_trigger_v2.go	85.22%	21 Missing and 9 partials :warning:
internal/datacoord/compaction_view.go	61.76%	11 Missing and 2 partials :warning:
internal/datacoord/compaction_policy.go	73.33%	6 Missing and 2 partials :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##              2.4   #35625      +/-   ##
==========================================
+ Coverage   80.23%   81.09%   +0.86%     
==========================================
  Files        1058     1057       -1     
  Lines      139433   139312     -121     
==========================================
+ Hits       111874   112980    +1106     
+ Misses      23277    22054    -1223     
+ Partials     4282     4278       -4     

Files with missing lines	Coverage Δ
internal/datacoord/compaction.go	80.69% <100.00%> (+1.72%)	:arrow_up:
internal/datacoord/compaction_policy_clustering.go	76.00% <100.00%> (+0.58%)	:arrow_up:
internal/datacoord/compaction_policy_l0.go	100.00% <100.00%> (ø)
internal/datacoord/compaction_trigger.go	87.31% <100.00%> (+1.23%)	:arrow_up:
internal/datacoord/services.go	85.09% <100.00%> (ø)
internal/datacoord/compaction_policy.go	73.33% <73.33%> (ø)
internal/datacoord/compaction_view.go	49.55% <61.76%> (-38.36%)	:arrow_down:
internal/datacoord/compaction_trigger_v2.go	79.16% <85.22%> (+5.27%)	:arrow_up:

... and 85 files with indirect coverage changes

[mergify[bot]]

@XuanYang-cn E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[XuanYang-cn]

/wip

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.