enhance: enable stream writer in compactions

[tedxu]

See #31679

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tedxu To complete the pull request process, please assign xiaofan-luan after the PR has been reviewed. You can assign the PR to them by writing /assign @xiaofan-luan in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• internal/datanode/OWNERS
• internal/storage/OWNERS

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

[mergify[bot]]

@tedxu

Invalid PR Title Format Detected

Your PR submission does not adhere to our required standards. To ensure clarity and consistency, please meet the following criteria:

1. Title Format: The PR title must begin with one of these prefixes:

• feat: for introducing a new feature.
• fix: for bug fixes.
• enhance: for improvements to existing functionality.
• test: for add tests to existing functionality.
• doc: for modifying documentation.
• auto: for the pull request from bot.

2. Description Requirement: The PR must include a non-empty description, detailing the changes and their impact.

Required Title Structure:

[Type]: [Description of the PR]

Where Type is one of feat, fix, enhance, test or doc.

Example:

enhance: improve search performance significantly 

Please review and update your PR to comply with these guidelines.

[tedxu]

In this PR, the stream writer (introduced in #31537) is integrated into the compactor. By removing the intermediate serialized data structure, the memory copy is greatly reduced. The benchmark (BenchmarkCompaction) showed a positive result:

Before:

BenchmarkCompaction-8         57           22303957 ns/op        251523736 B/op      8563 allocs/op
PASS

Enhanced:

BenchmarkCompaction-8         313           3407496 ns/op         1325833 B/op       6982 allocs/op
PASS

[mergify[bot]]

@tedxu E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify[bot]]

@tedxu E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[tedxu]

/run-cpu-e2e

[mergify[bot]]

@tedxu E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 79.22078% with 16 lines in your changes are missing coverage. Please review.

Project coverage is 82.11%. Comparing base (dd9919a) to head (3649770). Report is 13 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #32612      +/-   ##
==========================================
- Coverage   82.14%   82.11%   -0.04%     
==========================================
  Files        1006     1006              
  Lines      128194   128197       +3     
==========================================
- Hits       105306   105266      -40     
- Misses      18919    18954      +35     
- Partials     3969     3977       +8     

Files	Coverage Δ
internal/datanode/io/binlog_io.go	100.00% <ø> (ø)
internal/datanode/binlog_io.go	85.00% <66.66%> (-6.87%)	:arrow_down:
internal/storage/serde.go	88.96% <88.23%> (+1.72%)	:arrow_up:
internal/datanode/compactor.go	90.08% <77.77%> (-0.26%)	:arrow_down:

... and 34 files with indirect coverage changes

[tedxu]

/run-cpu-e2e

[mergify[bot]]

@tedxu E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[XuanYang-cn]

/lgtm

[mergify[bot]]

@tedxu E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[tedxu]

/run-cpu-e2e

[XuanYang-cn]

/lgtm

[congqixia]

/approve

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: congqixia, tedxu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• ~~internal/datanode/OWNERS~~ [congqixia]
• ~~internal/storage/OWNERS~~ [congqixia]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment