enhance: Remove pre-marking segments as L2 during clustering compaction

[xiaocai2333]

trafficstars

issue: #36686

This pr will remove pre-marking segments as L2 during clustering compaction in version 2.5, and ensure compatibility with version 2.4.

The core of this change is to ensure that the many-to-many lineage derivation logic is correct, making sure that both the parent and child cannot simultaneously exist in the target segment view.

feature:

• Clustering compaction no longer marks the input segments as L2.
• Add a new field is_invisible to segmentInfo, and mark segments that have completed clustering but have not yet built indexes as is_invisible to prevent them from being loaded prematurely."
• Do not mark the input segment as Dropped before the clustering compaction is completed.
• After compaction fails, only the result segment needs to be marked as Dropped.

compatibility:

• If the upgraded task has not failed, there are no compatibility issues.
  • If the status after the upgrade is MetaSaved, then skip the stats task based on whether TmpSegments is empty.
• If the failure occurs before MetaSaved:
  • there are no ResultSegments, and InputSegments have not been marked as dropped yet.
  • the level of input segments need to revert to LastLevel
• If the failure occurs after MetaSaved:
  • ResultSegments have already been generated, and InputSegments have been marked as Dropped. At this point, simply make the ResultSegments visible.
  • the level of ResultSegments needs to be set to L1（in order to participate in mixCompaction）

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 78.28571% with 38 lines in your changes missing coverage. Please review.

Project coverage is 80.68%. Comparing base (29910ff) to head (ce21a57). Report is 15 commits behind head on master.

:exclamation: Current head ce21a57 differs from pull request most recent head 8cb1a8d

Please upload reports for the commit 8cb1a8d to get more accurate results.

Files with missing lines	Patch %	Lines
internal/datacoord/compaction_task_clustering.go	63.95%	22 Missing and 9 partials :warning:
internal/datacoord/handler.go	86.36%	4 Missing and 2 partials :warning:
internal/datacoord/meta.go	97.67%	1 Missing :warning:

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #36799       +/-   ##
===========================================
+ Coverage   67.05%   80.68%   +13.63%     
===========================================
  Files         281     1310     +1029     
  Lines       27222   183609   +156387     
===========================================
+ Hits        18254   148151   +129897     
- Misses       8968    30331    +21363     
- Partials        0     5127     +5127     

Files with missing lines	Coverage Δ
internal/datacoord/partition_stats_meta.go	37.74% <100.00%> (ø)
internal/datacoord/meta.go	89.04% <97.67%> (ø)
internal/datacoord/handler.go	74.39% <86.36%> (ø)
internal/datacoord/compaction_task_clustering.go	76.14% <63.95%> (ø)

... and 1034 files with indirect coverage changes

[mergify[bot]]

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[xiaocai2333]

rerun ut

[xiaocai2333]

/run-cpu-e2e

[mergify[bot]]

@xiaocai2333 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[xiaocai2333]

/run-cpu-e2e

[czs007]

/approve /lgtm

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: czs007, xiaocai2333

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• ~~internal/datacoord/OWNERS~~ [czs007]
• ~~internal/proto/OWNERS~~ [czs007]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment