feat: add Storage V2 support to delta logs

[tedxu]

See: #39173

• Refactored delta log reading to accommodate both storage versions, improving flexibility and maintainability.
• Adjusted related components in clustering and mix compaction tasks to utilize the updated function signature.

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tedxu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• ~~OWNERS~~ [tedxu]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

[sre-ci-robot]

[ci-v2-notice] Notice: We are gradually rolling out the new ci-v2 system.

• Legacy CI jobs remain unaffected, you can just ignore ci-v2 if you don't want to run it.
• Additional "ci-v2/*" checkers will run for this PR to ensure the new ci-v2 system is working as expected.
• For tests that exist in both v1 and v2, passing in either system is considered PASS.

To rerun ci-v2 checks, comment with:

• /ci-rerun-code-check // for ci-v2/code-check
• /ci-rerun-build // for ci-v2/build
• /ci-rerun-ut-integration // for ci-v2/ut-integration
• /ci-rerun-ut-go // for ci-v2/ut-go
• /ci-rerun-ut-cpp // for ci-v2/ut-cpp
• /ci-rerun-ut // for all ci-v2/ut-integration, ci-v2/ut-go, ci-v2/ut-cpp
• /ci-rerun-e2e-arm // for ci-v2/e2e-arm

If you have any questions or requests, please contact @zhikunyao.

[mergify[bot]]

@tedxu Please associate the related issue to the body of your Pull Request. (eg. "issue: #")

[mergify[bot]]

@tedxu cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@tedxu go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify[bot]]

@tedxu cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[codecov[bot]]

Codecov Report

:x: Patch coverage is 77.48918% with 156 lines in your changes missing coverage. Please review. :white_check_mark: Project coverage is 76.49%. Comparing base (caed0fe) to head (09ed58c). :warning: Report is 17 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/datanode/compactor/l0_compactor.go	74.07%	20 Missing and 8 partials :warning:
internal/querynodev2/segments/segment_loader.go	61.29%	21 Missing and 3 partials :warning:
internal/storage/serde_delta.go	77.14%	12 Missing and 4 partials :warning:
internal/storage/delta_data.go	75.80%	13 Missing and 2 partials :warning:
internal/flushcommon/syncmgr/pack_writer.go	63.88%	10 Missing and 3 partials :warning:
internal/util/importutilv2/binlog/l0_reader.go	74.46%	10 Missing and 2 partials :warning:
internal/util/importutilv2/binlog/reader.go	77.55%	9 Missing and 2 partials :warning:
internal/compaction/common.go	85.07%	7 Missing and 3 partials :warning:
internal/flushcommon/syncmgr/pack_writer_v2.go	76.74%	7 Missing and 3 partials :warning:
internal/storage/serde_events.go	81.48%	6 Missing and 4 partials :warning:
... and 1 more

:x: Your patch check has failed because the patch coverage (77.48%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage. :x: Your project check has failed because the head coverage (76.49%) is below the target coverage (77.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #45279       +/-   ##
===========================================
- Coverage   83.18%   76.49%    -6.69%     
===========================================
  Files         521     1875     +1354     
  Lines       81313   292293   +210980     
===========================================
+ Hits        67642   223596   +155954     
- Misses      13671    61289    +47618     
- Partials        0     7408     +7408     

Components	Coverage Δ
Client	78.17% <ø> (∅)
Core	83.18% <ø> (ø)
Go	74.61% <77.48%> (∅)

Files with missing lines	Coverage Δ
...nternal/datanode/compactor/clustering_compactor.go	61.82% <100.00%> (ø)
internal/datanode/compactor/compactor_common.go	31.86% <ø> (ø)
internal/datanode/compactor/merge_sort.go	70.70% <100.00%> (ø)
internal/datanode/compactor/mix_compactor.go	70.97% <100.00%> (ø)
internal/datanode/compactor/segment_writer.go	62.11% <ø> (ø)
internal/datanode/compactor/sort_compaction.go	58.87% <100.00%> (ø)
internal/datanode/importv2/task_l0_import.go	72.45% <100.00%> (ø)
internal/datanode/importv2/task_l0_preimport.go	80.91% <100.00%> (ø)
internal/datanode/index/task_stats.go	52.83% <100.00%> (ø)
internal/querynodev2/segments/segment.go	58.82% <100.00%> (ø)
... and 12 more

... and 1332 files with indirect coverage changes

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mergify[bot]]

@tedxu go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify[bot]]

@tedxu cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@tedxu cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@tedxu cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@tedxu go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify[bot]]

@tedxu go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify[bot]]

@tedxu cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@tedxu go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify[bot]]

@tedxu cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@tedxu cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@tedxu go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify[bot]]

@tedxu cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@tedxu go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify[bot]]

@tedxu cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@tedxu go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify[bot]]

@tedxu cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@tedxu go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify[bot]]

@tedxu cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@tedxu cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[tedxu]

/run-cpu-e2e

[mergify[bot]]

@tedxu cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@tedxu cpu-e2e job failed, comment /run-cpu-e2e can trigger the job again.

[tedxu]

/ci-rerun-ut-integration